[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790243735634835-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 09:55:35 dist-test-slave-w8d2 krb5kdc[395](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 09:55:35 dist-test-slave-w8d2 krb5kdc[395](info): set up 2 sockets
Sep 24 09:55:35 dist-test-slave-w8d2 krb5kdc[395](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260924 09:55:36.679941 388 mini_kdc.cc:121] Time spent starting KDC: real 0.959s user 0.000s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790243735634835-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790243735634835-388-0/krb5kdc/test-user.keytab.
Sep 24 09:55:36 dist-test-slave-w8d2 krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243736, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T09:55:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T09:55:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790243735634835-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790243735634835-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260924 09:55:36.833221 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
/tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790243735634835-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790243735634835-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790243735634835-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790243735634835-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34257
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790243735634835-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:34257
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790243735634835-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790243735634835-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790243735634835-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 09:55:37.144289 408 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 09:55:37.144565 408 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 09:55:37.144696 408 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 09:55:37.151690 408 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 09:55:37.151790 408 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 09:55:37.151863 408 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 09:55:37.151932 408 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 09:55:37.151988 408 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 09:55:37.161554 408 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790243735634835-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790243735634835-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:34257
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790243735634835-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34257
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790243735634835-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790243735634835-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 09:55:37.162819 408 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 09:55:37.164131 408 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 09:55:37.175618 413 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:55:37.175590 414 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:55:37.175638 416 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 09:55:37.176180 408 server_base.cc:1034] running on GCE node
I20260924 09:55:37.177023 408 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 09:55:37.178741 408 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 09:55:37.179925 408 hybrid_clock.cc:648] HybridClock initialized: now 1790243737179871 us; error 47 us; skew 500 ppm
I20260924 09:55:37.180220 408 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 09:55:37 dist-test-slave-w8d2 krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243737, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 09:55:37.182996 408 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260924 09:55:37.184417 408 webserver.cc:460] Webserver started at http://127.0.97.62:40343/ using document root <none> and password file <none>
I20260924 09:55:37.185037 408 fs_manager.cc:359] Metadata directory not provided
I20260924 09:55:37.185127 408 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 09:55:37.185362 408 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 09:55:37.188264 408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790243735634835-388-0/minicluster-data/master-0/data/instance:
uuid: "1883802720c046b194a7f8984a6299ca"
format_stamp: "Formatted at 2026-09-24 09:55:37 on dist-test-slave-w8d2"
server_key: "1dfc22be273e865a1afe584af01f8e42"
server_key_iv: "f4e29242cbfd5ca8604dd9e6c6c985be"
server_key_version: "encryptionkey@0"
I20260924 09:55:37.188927 408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790243735634835-388-0/minicluster-data/master-0/wal/instance:
uuid: "1883802720c046b194a7f8984a6299ca"
format_stamp: "Formatted at 2026-09-24 09:55:37 on dist-test-slave-w8d2"
server_key: "1dfc22be273e865a1afe584af01f8e42"
server_key_iv: "f4e29242cbfd5ca8604dd9e6c6c985be"
server_key_version: "encryptionkey@0"
I20260924 09:55:37.193789 408 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 09:55:37.196962 423 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 09:55:37.198148 408 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 09:55:37.198290 408 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790243735634835-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790243735634835-388-0/minicluster-data/master-0/wal
uuid: "1883802720c046b194a7f8984a6299ca"
format_stamp: "Formatted at 2026-09-24 09:55:37 on dist-test-slave-w8d2"
server_key: "1dfc22be273e865a1afe584af01f8e42"
server_key_iv: "f4e29242cbfd5ca8604dd9e6c6c985be"
server_key_version: "encryptionkey@0"
I20260924 09:55:37.198446 408 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790243735634835-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790243735634835-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790243735634835-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 09:55:37.240607 408 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 09:55:37.244943 408 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 09:55:37.245235 408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 09:55:37.262465 408 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:34257
I20260924 09:55:37.262490 474 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:34257 every 8 connection(s)
I20260924 09:55:37.263793 408 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790243735634835-388-0/minicluster-data/master-0/data/info.pb
I20260924 09:55:37.267261 475 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 09:55:37.268563 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu as pid 408
I20260924 09:55:37.268867 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790243735634835-388-0/minicluster-data/master-0/wal/instance
I20260924 09:55:37.271138 388 external_mini_cluster.cc:1428] Setting key 37d608940d14ac7030d47260da35a468
I20260924 09:55:37.279313 475 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1883802720c046b194a7f8984a6299ca: Bootstrap starting.
I20260924 09:55:37.281917 475 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1883802720c046b194a7f8984a6299ca: Neither blocks nor log segments found. Creating new log.
I20260924 09:55:37.283175 475 log.cc:824] T 00000000000000000000000000000000 P 1883802720c046b194a7f8984a6299ca: Log is configured to *not* fsync() on all Append() calls
I20260924 09:55:37.286031 475 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1883802720c046b194a7f8984a6299ca: No bootstrap required, opened a new log
I20260924 09:55:37.294910 475 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1883802720c046b194a7f8984a6299ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1883802720c046b194a7f8984a6299ca" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34257 } }
I20260924 09:55:37.295260 475 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1883802720c046b194a7f8984a6299ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 09:55:37.295351 475 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1883802720c046b194a7f8984a6299ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1883802720c046b194a7f8984a6299ca, State: Initialized, Role: FOLLOWER
I20260924 09:55:37.295858 475 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1883802720c046b194a7f8984a6299ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1883802720c046b194a7f8984a6299ca" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34257 } }
I20260924 09:55:37.296096 475 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1883802720c046b194a7f8984a6299ca [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 09:55:37.296234 475 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1883802720c046b194a7f8984a6299ca [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 09:55:37.296378 475 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1883802720c046b194a7f8984a6299ca [term 0 FOLLOWER]: Advancing to term 1
I20260924 09:55:37.298331 475 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1883802720c046b194a7f8984a6299ca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1883802720c046b194a7f8984a6299ca" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34257 } }
Sep 24 09:55:37 dist-test-slave-w8d2 krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243736, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 09:55:37.298756 475 leader_election.cc:302] T 00000000000000000000000000000000 P 1883802720c046b194a7f8984a6299ca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1883802720c046b194a7f8984a6299ca; no voters:
I20260924 09:55:37.299310 475 leader_election.cc:288] T 00000000000000000000000000000000 P 1883802720c046b194a7f8984a6299ca [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 09:55:37.299468 480 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1883802720c046b194a7f8984a6299ca [term 1 FOLLOWER]: Leader election won for term 1
I20260924 09:55:37.301142 475 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1883802720c046b194a7f8984a6299ca [sys.catalog]: configured and running, proceeding with master startup.
I20260924 09:55:37.302928 480 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1883802720c046b194a7f8984a6299ca [term 1 LEADER]: Becoming Leader. State: Replica: 1883802720c046b194a7f8984a6299ca, State: Running, Role: LEADER
I20260924 09:55:37.303846 480 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1883802720c046b194a7f8984a6299ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1883802720c046b194a7f8984a6299ca" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34257 } }
I20260924 09:55:37.306749 481 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1883802720c046b194a7f8984a6299ca [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1883802720c046b194a7f8984a6299ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1883802720c046b194a7f8984a6299ca" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34257 } } }
I20260924 09:55:37.307385 481 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1883802720c046b194a7f8984a6299ca [sys.catalog]: This master's current role is: LEADER
I20260924 09:55:37.307019 482 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1883802720c046b194a7f8984a6299ca [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1883802720c046b194a7f8984a6299ca. Latest consensus state: current_term: 1 leader_uuid: "1883802720c046b194a7f8984a6299ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1883802720c046b194a7f8984a6299ca" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34257 } } }
I20260924 09:55:37.307986 482 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1883802720c046b194a7f8984a6299ca [sys.catalog]: This master's current role is: LEADER
I20260924 09:55:37.312745 477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:55:37.275453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37206 (local address 127.0.97.62:34257)
0924 09:55:37.276642 (+ 1189us) server_negotiation.cc:206] Beginning negotiation
0924 09:55:37.276656 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 09:55:37.277842 (+ 1186us) server_negotiation.cc:407] Connection header received
0924 09:55:37.286742 (+ 8900us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:55:37.286768 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:55:37.287204 (+ 436us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:55:37.287614 (+ 410us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:55:37.290116 (+ 2502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:55:37.292059 (+ 1943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:55:37.293610 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:55:37.294582 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:55:37.294753 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:55:37.300262 (+ 5509us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:55:37.300273 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:55:37.300293 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:55:37.300341 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:55:37.305531 (+ 5190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:55:37.306502 (+ 971us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:55:37.306512 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:55:37.306520 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:55:37.306748 (+ 228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:55:37.307319 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:55:37.307323 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:55:37.307326 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:55:37.308066 (+ 740us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:55:37.308371 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0924 09:55:37.310918 (+ 2547us) server_negotiation.cc:299] Negotiation successful
0924 09:55:37.311388 (+ 470us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":948,"thread_start_us":730,"threads_started":1}
I20260924 09:55:37.320654 489 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 09:55:37.325497 489 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 09:55:37.334733 489 catalog_manager.cc:1347] Generated new cluster ID: eacddf80e965449aa3d2cb12767ffbbe
I20260924 09:55:37.334852 489 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 09:55:37.366874 489 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 09:55:37.368716 489 catalog_manager.cc:1504] Loading token signing keys...
I20260924 09:55:37.376139 489 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1883802720c046b194a7f8984a6299ca: Generated new TSK 0
I20260924 09:55:37.376905 489 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790243735634835-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790243735634835-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260924 09:55:37.432651 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
/tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790243735634835-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790243735634835-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790243735634835-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790243735634835-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34257
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790243735634835-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790243735634835-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790243735634835-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790243735634835-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 09:55:37.758697 501 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 09:55:37.759002 501 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 09:55:37.759140 501 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 09:55:37.767084 501 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 09:55:37.767199 501 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 09:55:37.767364 501 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260924 09:55:37.776973 501 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790243735634835-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790243735634835-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790243735634835-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790243735634835-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34257
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790243735634835-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 09:55:37.778179 501 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 09:55:37.779477 501 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 09:55:37.792095 506 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:55:37.792240 507 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:55:37.792270 509 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 09:55:37.792698 501 server_base.cc:1034] running on GCE node
I20260924 09:55:37.793344 501 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 09:55:37.794224 501 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 09:55:37.795400 501 hybrid_clock.cc:648] HybridClock initialized: now 1790243737795351 us; error 44 us; skew 500 ppm
I20260924 09:55:37.795730 501 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 09:55:37 dist-test-slave-w8d2 krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243737, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 09:55:37.798899 501 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260924 09:55:37.800216 501 webserver.cc:460] Webserver started at http://127.0.97.1:38083/ using document root <none> and password file <none>
I20260924 09:55:37.800869 501 fs_manager.cc:359] Metadata directory not provided
I20260924 09:55:37.800959 501 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 09:55:37.801229 501 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 09:55:37.804051 501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790243735634835-388-0/minicluster-data/ts-0/data/instance:
uuid: "c8ecdb60630b452e9477922472f3b6be"
format_stamp: "Formatted at 2026-09-24 09:55:37 on dist-test-slave-w8d2"
server_key: "7e45f6cab5854f0864e801a98fe839e5"
server_key_iv: "8a01cf045147492d7ab0fc3419c5fa79"
server_key_version: "encryptionkey@0"
I20260924 09:55:37.804757 501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790243735634835-388-0/minicluster-data/ts-0/wal/instance:
uuid: "c8ecdb60630b452e9477922472f3b6be"
format_stamp: "Formatted at 2026-09-24 09:55:37 on dist-test-slave-w8d2"
server_key: "7e45f6cab5854f0864e801a98fe839e5"
server_key_iv: "8a01cf045147492d7ab0fc3419c5fa79"
server_key_version: "encryptionkey@0"
I20260924 09:55:37.809533 501 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 09:55:37.813051 516 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 09:55:37.814301 501 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 09:55:37.814419 501 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790243735634835-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790243735634835-388-0/minicluster-data/ts-0/wal
uuid: "c8ecdb60630b452e9477922472f3b6be"
format_stamp: "Formatted at 2026-09-24 09:55:37 on dist-test-slave-w8d2"
server_key: "7e45f6cab5854f0864e801a98fe839e5"
server_key_iv: "8a01cf045147492d7ab0fc3419c5fa79"
server_key_version: "encryptionkey@0"
I20260924 09:55:37.814567 501 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790243735634835-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790243735634835-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790243735634835-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 09:55:37.836879 501 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 09:55:37.840620 501 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 09:55:37.840852 501 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 09:55:37.841704 501 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 09:55:37.843005 501 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 09:55:37.843075 501 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 09:55:37.843163 501 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 09:55:37.843204 501 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 09:55:37.873626 501 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:42095
I20260924 09:55:37.873657 628 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:42095 every 8 connection(s)
I20260924 09:55:37.874838 501 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790243735634835-388-0/minicluster-data/ts-0/data/info.pb
I20260924 09:55:37.878614 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu as pid 501
I20260924 09:55:37.878819 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790243735634835-388-0/minicluster-data/ts-0/wal/instance
I20260924 09:55:37.879393 388 external_mini_cluster.cc:1428] Setting key 546fdce09faf65224ec22b83a5c213cf
Sep 24 09:55:37 dist-test-slave-w8d2 krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243737, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260924 09:55:37.895381 631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:55:37.877208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:54603 (local address 127.0.97.62:34257)
0924 09:55:37.877609 (+ 401us) server_negotiation.cc:206] Beginning negotiation
0924 09:55:37.877615 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 09:55:37.878563 (+ 948us) server_negotiation.cc:407] Connection header received
0924 09:55:37.881081 (+ 2518us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:55:37.881086 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:55:37.881159 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:55:37.881255 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:55:37.883249 (+ 1994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:55:37.883924 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:55:37.885840 (+ 1916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:55:37.886428 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:55:37.886548 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:55:37.890054 (+ 3506us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:55:37.890061 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:55:37.890066 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:55:37.890111 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:55:37.892425 (+ 2314us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:55:37.893224 (+ 799us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:55:37.893228 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:55:37.893231 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:55:37.893335 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:55:37.893767 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:55:37.893770 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:55:37.893772 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:55:37.894017 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:55:37.894130 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0924 09:55:37.894889 (+ 759us) server_negotiation.cc:299] Negotiation successful
0924 09:55:37.895060 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":189,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260924 09:55:37.896324 632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:55:37.877721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34257 (local address 127.0.97.1:54603)
0924 09:55:37.878411 (+ 690us) negotiation.cc:107] Waiting for socket to connect
0924 09:55:37.878453 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 09:55:37.880693 (+ 2240us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:55:37.881516 (+ 823us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:55:37.881526 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:55:37.882181 (+ 655us) client_negotiation.cc:189] Negotiated authn=SASL
0924 09:55:37.883088 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:55:37.883111 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:55:37.884055 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:55:37.884060 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:55:37.885723 (+ 1663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:55:37.885734 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:55:37.886672 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:55:37.886678 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:55:37.886866 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:55:37.887589 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 09:55:37.887617 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0924 09:55:37.889827 (+ 2210us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 09:55:37.892551 (+ 2724us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 09:55:37.892561 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 09:55:37.892580 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0924 09:55:37.893121 (+ 541us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 09:55:37.893429 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 09:55:37.893435 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 09:55:37.893438 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 09:55:37.893661 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 09:55:37.894107 (+ 446us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 09:55:37.894113 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 09:55:37.894464 (+ 351us) client_negotiation.cc:769] Sending connection context
0924 09:55:37.894703 (+ 239us) client_negotiation.cc:240] Negotiation successful
0924 09:55:37.895074 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":251,"threads_started":1}
I20260924 09:55:37.897758 629 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34257
I20260924 09:55:37.898090 629 heartbeater.cc:464] Registering TS with master...
I20260924 09:55:37.898855 629 heartbeater.cc:511] Master 127.0.97.62:34257 requested a full tablet report, sending...
I20260924 09:55:37.900898 440 ts_manager.cc:194] Registered new tserver with Master: c8ecdb60630b452e9477922472f3b6be (127.0.97.1:42095)
I20260924 09:55:37.902729 440 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:54603
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790243735634835-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790243735634835-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260924 09:55:37.915343 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
/tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790243735634835-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790243735634835-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790243735634835-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790243735634835-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34257
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790243735634835-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790243735634835-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790243735634835-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790243735634835-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 09:55:38.231972 635 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 09:55:38.232251 635 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 09:55:38.232344 635 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 09:55:38.241812 635 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 09:55:38.241931 635 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 09:55:38.242045 635 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260924 09:55:38.252172 635 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790243735634835-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790243735634835-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790243735634835-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790243735634835-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34257
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790243735634835-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 09:55:38.253389 635 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 09:55:38.254665 635 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 09:55:38.267196 640 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:55:38.267931 643 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:55:38.267956 641 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 09:55:38.269531 635 server_base.cc:1034] running on GCE node
I20260924 09:55:38.270123 635 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 09:55:38.271013 635 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 09:55:38.272235 635 hybrid_clock.cc:648] HybridClock initialized: now 1790243738272175 us; error 50 us; skew 500 ppm
I20260924 09:55:38.272548 635 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 09:55:38 dist-test-slave-w8d2 krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243738, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 09:55:38.279726 635 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260924 09:55:38.281522 635 webserver.cc:460] Webserver started at http://127.0.97.2:46235/ using document root <none> and password file <none>
I20260924 09:55:38.282438 635 fs_manager.cc:359] Metadata directory not provided
I20260924 09:55:38.282574 635 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 09:55:38.282871 635 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 09:55:38.285991 635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790243735634835-388-0/minicluster-data/ts-1/data/instance:
uuid: "4130eb84864b44b1a096a41e66ef4836"
format_stamp: "Formatted at 2026-09-24 09:55:38 on dist-test-slave-w8d2"
server_key: "71af53995b856318ccf27e0832ef1139"
server_key_iv: "24b5e27c0f8fac578614f219b9dfc9f1"
server_key_version: "encryptionkey@0"
I20260924 09:55:38.286947 635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790243735634835-388-0/minicluster-data/ts-1/wal/instance:
uuid: "4130eb84864b44b1a096a41e66ef4836"
format_stamp: "Formatted at 2026-09-24 09:55:38 on dist-test-slave-w8d2"
server_key: "71af53995b856318ccf27e0832ef1139"
server_key_iv: "24b5e27c0f8fac578614f219b9dfc9f1"
server_key_version: "encryptionkey@0"
I20260924 09:55:38.292248 635 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.000s
I20260924 09:55:38.295343 650 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 09:55:38.296520 635 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260924 09:55:38.296654 635 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790243735634835-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790243735634835-388-0/minicluster-data/ts-1/wal
uuid: "4130eb84864b44b1a096a41e66ef4836"
format_stamp: "Formatted at 2026-09-24 09:55:38 on dist-test-slave-w8d2"
server_key: "71af53995b856318ccf27e0832ef1139"
server_key_iv: "24b5e27c0f8fac578614f219b9dfc9f1"
server_key_version: "encryptionkey@0"
I20260924 09:55:38.296832 635 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790243735634835-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790243735634835-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790243735634835-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 09:55:38.315397 635 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 09:55:38.319579 635 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 09:55:38.319854 635 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 09:55:38.320762 635 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 09:55:38.322041 635 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 09:55:38.322115 635 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 09:55:38.322230 635 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 09:55:38.322281 635 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 09:55:38.353847 635 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:46775
I20260924 09:55:38.353888 762 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:46775 every 8 connection(s)
I20260924 09:55:38.355070 635 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790243735634835-388-0/minicluster-data/ts-1/data/info.pb
I20260924 09:55:38.361405 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu as pid 635
I20260924 09:55:38.361567 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790243735634835-388-0/minicluster-data/ts-1/wal/instance
I20260924 09:55:38.361953 388 external_mini_cluster.cc:1428] Setting key 5b8579b371af4932e6d8542218c53b13
Sep 24 09:55:38 dist-test-slave-w8d2 krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243738, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 09:55:38.374226 631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:55:38.357496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:55377 (local address 127.0.97.62:34257)
0924 09:55:38.357657 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0924 09:55:38.357661 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 09:55:38.358707 (+ 1046us) server_negotiation.cc:407] Connection header received
0924 09:55:38.360428 (+ 1721us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:55:38.360432 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:55:38.360499 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:55:38.360573 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:55:38.362011 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:55:38.362825 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:55:38.364459 (+ 1634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:55:38.364856 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:55:38.364960 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:55:38.368791 (+ 3831us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:55:38.368796 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:55:38.368799 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:55:38.368830 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:55:38.371272 (+ 2442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:55:38.372073 (+ 801us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:55:38.372076 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:55:38.372079 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:55:38.372198 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:55:38.372667 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:55:38.372672 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:55:38.372679 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:55:38.372936 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:55:38.373037 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0924 09:55:38.373842 (+ 805us) server_negotiation.cc:299] Negotiation successful
0924 09:55:38.373999 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260924 09:55:38.375721 765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:55:38.357898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34257 (local address 127.0.97.2:55377)
0924 09:55:38.358543 (+ 645us) negotiation.cc:107] Waiting for socket to connect
0924 09:55:38.358584 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 09:55:38.360173 (+ 1589us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:55:38.360779 (+ 606us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:55:38.360786 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:55:38.361256 (+ 470us) client_negotiation.cc:189] Negotiated authn=SASL
0924 09:55:38.361851 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:55:38.361869 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:55:38.362954 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:55:38.362957 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:55:38.364335 (+ 1378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:55:38.364345 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:55:38.365055 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:55:38.365062 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:55:38.365267 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:55:38.365901 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 09:55:38.365926 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0924 09:55:38.368582 (+ 2656us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 09:55:38.371386 (+ 2804us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 09:55:38.371395 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 09:55:38.371412 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 09:55:38.371962 (+ 550us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 09:55:38.372297 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 09:55:38.372302 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 09:55:38.372306 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 09:55:38.372549 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 09:55:38.373049 (+ 500us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 09:55:38.373057 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 09:55:38.373495 (+ 438us) client_negotiation.cc:769] Sending connection context
0924 09:55:38.373793 (+ 298us) client_negotiation.cc:240] Negotiation successful
0924 09:55:38.374336 (+ 543us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":237,"threads_started":1}
I20260924 09:55:38.377139 763 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34257
I20260924 09:55:38.377403 763 heartbeater.cc:464] Registering TS with master...
I20260924 09:55:38.378098 763 heartbeater.cc:511] Master 127.0.97.62:34257 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260924 09:55:38.379614 440 ts_manager.cc:194] Registered new tserver with Master: 4130eb84864b44b1a096a41e66ef4836 (127.0.97.2:46775)
I20260924 09:55:38.380628 440 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:55377
I20260924 09:55:38.905503 629 heartbeater.cc:503] Master 127.0.97.62:34257 was elected leader, sending a full tablet report...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790243735634835-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790243735634835-388-0/krb5kdc/kudu_127.0.97.3.keytab.
W20260924 09:55:39.073267 388 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.97.3: real 0.709s user 0.001s sys 0.006s
I20260924 09:55:39.073973 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
/tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790243735634835-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790243735634835-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790243735634835-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790243735634835-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34257
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790243735634835-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790243735634835-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790243735634835-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790243735634835-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 09:55:39.383092 763 heartbeater.cc:503] Master 127.0.97.62:34257 was elected leader, sending a full tablet report...
W20260924 09:55:39.397385 768 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 09:55:39.397666 768 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 09:55:39.397807 768 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 09:55:39.405180 768 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 09:55:39.405297 768 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 09:55:39.405444 768 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260924 09:55:39.415621 768 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790243735634835-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790243735634835-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790243735634835-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790243735634835-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34257
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790243735634835-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 09:55:39.416867 768 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 09:55:39.418213 768 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 09:55:39.430707 773 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:55:39.430778 774 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:55:39.430811 776 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 09:55:39.432667 768 server_base.cc:1034] running on GCE node
I20260924 09:55:39.433228 768 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 09:55:39.434053 768 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 09:55:39.435258 768 hybrid_clock.cc:648] HybridClock initialized: now 1790243739435188 us; error 44 us; skew 500 ppm
I20260924 09:55:39.435559 768 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 09:55:39 dist-test-slave-w8d2 krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243739, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 09:55:39.438503 768 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260924 09:55:39.439886 768 webserver.cc:460] Webserver started at http://127.0.97.3:41741/ using document root <none> and password file <none>
I20260924 09:55:39.440528 768 fs_manager.cc:359] Metadata directory not provided
I20260924 09:55:39.440625 768 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 09:55:39.440907 768 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 09:55:39.443651 768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790243735634835-388-0/minicluster-data/ts-2/data/instance:
uuid: "942d0869b98a417dbf53a76e559b5541"
format_stamp: "Formatted at 2026-09-24 09:55:39 on dist-test-slave-w8d2"
server_key: "6cd7fc034d56050051aa8cd70a0bbeb3"
server_key_iv: "cf8d169ca60575f0edbd505262e1344a"
server_key_version: "encryptionkey@0"
I20260924 09:55:39.444360 768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790243735634835-388-0/minicluster-data/ts-2/wal/instance:
uuid: "942d0869b98a417dbf53a76e559b5541"
format_stamp: "Formatted at 2026-09-24 09:55:39 on dist-test-slave-w8d2"
server_key: "6cd7fc034d56050051aa8cd70a0bbeb3"
server_key_iv: "cf8d169ca60575f0edbd505262e1344a"
server_key_version: "encryptionkey@0"
I20260924 09:55:39.449371 768 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 09:55:39.452617 783 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 09:55:39.453836 768 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260924 09:55:39.453987 768 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790243735634835-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790243735634835-388-0/minicluster-data/ts-2/wal
uuid: "942d0869b98a417dbf53a76e559b5541"
format_stamp: "Formatted at 2026-09-24 09:55:39 on dist-test-slave-w8d2"
server_key: "6cd7fc034d56050051aa8cd70a0bbeb3"
server_key_iv: "cf8d169ca60575f0edbd505262e1344a"
server_key_version: "encryptionkey@0"
I20260924 09:55:39.454161 768 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790243735634835-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790243735634835-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790243735634835-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 09:55:39.482952 768 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 09:55:39.486723 768 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 09:55:39.486984 768 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 09:55:39.487968 768 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 09:55:39.489192 768 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 09:55:39.489267 768 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 09:55:39.489387 768 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 09:55:39.489436 768 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 09:55:39.521070 768 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:44685
I20260924 09:55:39.521107 895 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:44685 every 8 connection(s)
I20260924 09:55:39.522270 768 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790243735634835-388-0/minicluster-data/ts-2/data/info.pb
I20260924 09:55:39.528294 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu as pid 768
I20260924 09:55:39.528432 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790243735634835-388-0/minicluster-data/ts-2/wal/instance
I20260924 09:55:39.528824 388 external_mini_cluster.cc:1428] Setting key 46fdd629677c2f2a7b80a6fd20219499
Sep 24 09:55:39 dist-test-slave-w8d2 krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243739, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 09:55:39.541015 898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:55:39.524602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:52305 (local address 127.0.97.62:34257)
0924 09:55:39.525415 (+ 813us) server_negotiation.cc:206] Beginning negotiation
0924 09:55:39.525421 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 09:55:39.526171 (+ 750us) server_negotiation.cc:407] Connection header received
0924 09:55:39.528015 (+ 1844us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:55:39.528020 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:55:39.528097 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:55:39.528190 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:55:39.529783 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:55:39.530798 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:55:39.532584 (+ 1786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:55:39.532966 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:55:39.533076 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:55:39.536224 (+ 3148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:55:39.536229 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:55:39.536232 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:55:39.536261 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:55:39.538549 (+ 2288us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:55:39.539163 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:55:39.539167 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:55:39.539169 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:55:39.539311 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:55:39.539647 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:55:39.539650 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:55:39.539652 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:55:39.539891 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:55:39.539999 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0924 09:55:39.540575 (+ 576us) server_negotiation.cc:299] Negotiation successful
0924 09:55:39.540736 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":709,"thread_start_us":589,"threads_started":1}
I20260924 09:55:39.542053 899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:55:39.525242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34257 (local address 127.0.97.3:52305)
0924 09:55:39.525997 (+ 755us) negotiation.cc:107] Waiting for socket to connect
0924 09:55:39.526034 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 09:55:39.527746 (+ 1712us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:55:39.528456 (+ 710us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:55:39.528465 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:55:39.529011 (+ 546us) client_negotiation.cc:189] Negotiated authn=SASL
0924 09:55:39.529615 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:55:39.529641 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:55:39.530936 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:55:39.530940 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:55:39.532464 (+ 1524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:55:39.532476 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:55:39.533169 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:55:39.533174 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:55:39.533382 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:55:39.533946 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 09:55:39.533973 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0924 09:55:39.536055 (+ 2082us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 09:55:39.538661 (+ 2606us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 09:55:39.538668 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 09:55:39.538679 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0924 09:55:39.539073 (+ 394us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 09:55:39.539397 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 09:55:39.539400 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 09:55:39.539402 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 09:55:39.539567 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 09:55:39.539978 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 09:55:39.539984 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 09:55:39.540278 (+ 294us) client_negotiation.cc:769] Sending connection context
0924 09:55:39.540510 (+ 232us) client_negotiation.cc:240] Negotiation successful
0924 09:55:39.540895 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":322,"threads_started":1}
I20260924 09:55:39.543520 896 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34257
I20260924 09:55:39.543793 896 heartbeater.cc:464] Registering TS with master...
I20260924 09:55:39.544479 896 heartbeater.cc:511] Master 127.0.97.62:34257 requested a full tablet report, sending...
I20260924 09:55:39.545792 440 ts_manager.cc:194] Registered new tserver with Master: 942d0869b98a417dbf53a76e559b5541 (127.0.97.3:44685)
I20260924 09:55:39.546587 440 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:52305
I20260924 09:55:39.556468 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 09:55:39 dist-test-slave-w8d2 krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243739, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 09:55:39 dist-test-slave-w8d2 krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243739, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260924 09:55:39.674336 904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:55:39.659782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53444 (local address 127.0.97.1:42095)
0924 09:55:39.660560 (+ 778us) server_negotiation.cc:206] Beginning negotiation
0924 09:55:39.660568 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 09:55:39.660599 (+ 31us) server_negotiation.cc:407] Connection header received
0924 09:55:39.660687 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:55:39.660696 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:55:39.660969 (+ 273us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:55:39.661132 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:55:39.662382 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:55:39.663408 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:55:39.664722 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:55:39.665250 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:55:39.665340 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:55:39.669273 (+ 3933us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:55:39.669285 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:55:39.669300 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:55:39.669335 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:55:39.672006 (+ 2671us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:55:39.672598 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:55:39.672605 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:55:39.672608 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:55:39.672715 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:55:39.673024 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:55:39.673027 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:55:39.673029 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:55:39.673323 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:55:39.673534 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0924 09:55:39.673820 (+ 286us) server_negotiation.cc:299] Negotiation successful
0924 09:55:39.674001 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":610,"thread_start_us":464,"threads_started":1}
W20260924 09:55:39.675557 544 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53444
Sep 24 09:55:39 dist-test-slave-w8d2 krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243739, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 09:55:39 dist-test-slave-w8d2 krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243739, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260924 09:55:39.773418 904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:55:39.759361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53450 (local address 127.0.97.1:42095)
0924 09:55:39.759548 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0924 09:55:39.759554 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 09:55:39.761000 (+ 1446us) server_negotiation.cc:407] Connection header received
0924 09:55:39.761176 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:55:39.761182 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:55:39.761291 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:55:39.761385 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:55:39.762874 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:55:39.763795 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:55:39.765008 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:55:39.765415 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:55:39.765526 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:55:39.769002 (+ 3476us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:55:39.769007 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:55:39.769011 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:55:39.769042 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:55:39.771158 (+ 2116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:55:39.771746 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:55:39.771751 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:55:39.771755 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:55:39.771882 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:55:39.772204 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:55:39.772209 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:55:39.772213 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:55:39.772436 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:55:39.772640 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0924 09:55:39.772920 (+ 280us) server_negotiation.cc:299] Negotiation successful
0924 09:55:39.773113 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260924 09:55:39.774833 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu pid 501
I20260924 09:55:39.786274 904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:55:39.775694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53454 (local address 127.0.97.1:42095)
0924 09:55:39.775825 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0924 09:55:39.775829 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 09:55:39.776185 (+ 356us) server_negotiation.cc:407] Connection header received
0924 09:55:39.776289 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:55:39.776292 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:55:39.776341 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:55:39.776409 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:55:39.777414 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:55:39.778064 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:55:39.779217 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:55:39.779648 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:55:39.779733 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:55:39.781635 (+ 1902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:55:39.781639 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:55:39.781642 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:55:39.781664 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:55:39.784457 (+ 2793us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:55:39.785004 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:55:39.785007 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:55:39.785009 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:55:39.785126 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:55:39.785359 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:55:39.785361 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:55:39.785363 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:55:39.785583 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:55:39.785743 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0924 09:55:39.785904 (+ 161us) server_negotiation.cc:299] Negotiation successful
0924 09:55:39.786035 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 09:55:39.786903 624 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 09:55:39.869091 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu with pid 501
I20260924 09:55:39.882385 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu pid 635
Sep 24 09:55:39 dist-test-slave-w8d2 krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243739, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260924 09:55:39.897239 911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:55:39.883284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36478 (local address 127.0.97.2:46775)
0924 09:55:39.883885 (+ 601us) server_negotiation.cc:206] Beginning negotiation
0924 09:55:39.883893 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 09:55:39.883918 (+ 25us) server_negotiation.cc:407] Connection header received
0924 09:55:39.883975 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:55:39.883981 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:55:39.884213 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:55:39.884353 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:55:39.885468 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:55:39.886467 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:55:39.887721 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:55:39.888280 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:55:39.888391 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:55:39.892072 (+ 3681us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:55:39.892084 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:55:39.892100 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:55:39.892130 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:55:39.894589 (+ 2459us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:55:39.895187 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:55:39.895194 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:55:39.895196 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:55:39.895415 (+ 219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:55:39.895793 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:55:39.895798 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:55:39.895800 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:55:39.896114 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:55:39.896392 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0924 09:55:39.896655 (+ 263us) server_negotiation.cc:299] Negotiation successful
0924 09:55:39.896897 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":462,"thread_start_us":300,"threads_started":1}
I20260924 09:55:39.898542 758 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 09:55:39.979316 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu with pid 635
I20260924 09:55:39.997465 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu pid 768
Sep 24 09:55:40 dist-test-slave-w8d2 krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243739, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260924 09:55:40.013327 913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:55:39.998358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52450 (local address 127.0.97.3:44685)
0924 09:55:39.998860 (+ 502us) server_negotiation.cc:206] Beginning negotiation
0924 09:55:39.998868 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 09:55:39.998918 (+ 50us) server_negotiation.cc:407] Connection header received
0924 09:55:39.998977 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:55:39.998983 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:55:39.999187 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:55:39.999345 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:55:40.000427 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:55:40.001429 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:55:40.002670 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:55:40.003439 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:55:40.003532 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:55:40.007775 (+ 4243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:55:40.007790 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:55:40.007810 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:55:40.007855 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:55:40.010621 (+ 2766us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:55:40.011346 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:55:40.011354 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:55:40.011356 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:55:40.011477 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:55:40.011811 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:55:40.011816 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:55:40.011830 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:55:40.012151 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:55:40.012431 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0924 09:55:40.012741 (+ 310us) server_negotiation.cc:299] Negotiation successful
0924 09:55:40.012995 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363,"thread_start_us":235,"threads_started":1}
I20260924 09:55:40.014590 891 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 09:55:40.095032 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu with pid 768
I20260924 09:55:40.108999 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu pid 408
I20260924 09:55:40.110105 470 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 09:55:40.213279 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu with pid 408
2026-09-24T09:55:40Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (4539 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790243735634835-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 09:55:41 dist-test-slave-w8d2 krb5kdc[918](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 09:55:41 dist-test-slave-w8d2 krb5kdc[918](info): set up 2 sockets
Sep 24 09:55:41 dist-test-slave-w8d2 krb5kdc[918](info): commencing operation
krb5kdc: starting...
W20260924 09:55:42.667503 388 mini_kdc.cc:121] Time spent starting KDC: real 2.417s user 0.006s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260924 09:55:42.813421 388 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.146s user 0.001s sys 0.005s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
W20260924 09:55:43.051340 388 mini_kdc.cc:253] Time spent creating user principal test-user: real 0.238s user 0.001s sys 0.002s
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
W20260924 09:55:43.344564 388 mini_kdc.cc:253] Time spent creating user principal joe-interloper: real 0.293s user 0.003s sys 0.002s
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790243735634835-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790243735634835-388-0/krb5kdc/test-user.keytab.
W20260924 09:55:43.928596 388 mini_kdc.cc:302] Time spent creating keytab for test-user: real 0.584s user 0.000s sys 0.004s
Sep 24 09:55:43 dist-test-slave-w8d2 krb5kdc[918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243743, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T09:55:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T09:55:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790243735634835-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790243735634835-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260924 09:55:44.028810 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
/tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790243735634835-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790243735634835-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790243735634835-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790243735634835-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40259
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790243735634835-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:40259
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790243735634835-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790243735634835-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790243735634835-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 09:55:44.328663 931 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 09:55:44.328969 931 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 09:55:44.329089 931 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 09:55:44.336171 931 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 09:55:44.336283 931 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 09:55:44.336328 931 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 09:55:44.336378 931 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 09:55:44.336436 931 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 09:55:44.346200 931 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790243735634835-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790243735634835-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:40259
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790243735634835-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40259
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790243735634835-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790243735634835-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 09:55:44.347519 931 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 09:55:44.348778 931 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 09:55:44.359289 936 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:55:44.359364 937 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:55:44.359380 939 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 09:55:44.360075 931 server_base.cc:1034] running on GCE node
I20260924 09:55:44.360744 931 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 09:55:44.362102 931 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 09:55:44.363310 931 hybrid_clock.cc:648] HybridClock initialized: now 1790243744363267 us; error 26 us; skew 500 ppm
I20260924 09:55:44.363617 931 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 09:55:44 dist-test-slave-w8d2 krb5kdc[918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243744, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 09:55:44.369381 931 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260924 09:55:44.370707 931 webserver.cc:460] Webserver started at http://127.0.97.62:34951/ using document root <none> and password file <none>
I20260924 09:55:44.371380 931 fs_manager.cc:359] Metadata directory not provided
I20260924 09:55:44.371505 931 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 09:55:44.371769 931 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 09:55:44.374516 931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790243735634835-388-0/minicluster-data/master-0/data/instance:
uuid: "e74c3f5132fb454dadecb6e932fc3c5f"
format_stamp: "Formatted at 2026-09-24 09:55:44 on dist-test-slave-w8d2"
server_key: "7019cf8d5a10e7dc5984b02611083cd7"
server_key_iv: "66cd01693817778b313f9c49eebd82b9"
server_key_version: "encryptionkey@0"
I20260924 09:55:44.375202 931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790243735634835-388-0/minicluster-data/master-0/wal/instance:
uuid: "e74c3f5132fb454dadecb6e932fc3c5f"
format_stamp: "Formatted at 2026-09-24 09:55:44 on dist-test-slave-w8d2"
server_key: "7019cf8d5a10e7dc5984b02611083cd7"
server_key_iv: "66cd01693817778b313f9c49eebd82b9"
server_key_version: "encryptionkey@0"
I20260924 09:55:44.380090 931 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.000s
I20260924 09:55:44.383327 946 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 09:55:44.384563 931 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260924 09:55:44.384713 931 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790243735634835-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790243735634835-388-0/minicluster-data/master-0/wal
uuid: "e74c3f5132fb454dadecb6e932fc3c5f"
format_stamp: "Formatted at 2026-09-24 09:55:44 on dist-test-slave-w8d2"
server_key: "7019cf8d5a10e7dc5984b02611083cd7"
server_key_iv: "66cd01693817778b313f9c49eebd82b9"
server_key_version: "encryptionkey@0"
I20260924 09:55:44.384889 931 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790243735634835-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790243735634835-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790243735634835-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 09:55:44.398458 931 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 09:55:44.402211 931 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 09:55:44.402454 931 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 09:55:44.418465 931 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:40259
I20260924 09:55:44.418485 997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:40259 every 8 connection(s)
I20260924 09:55:44.419813 931 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790243735634835-388-0/minicluster-data/master-0/data/info.pb
I20260924 09:55:44.422610 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu as pid 931
I20260924 09:55:44.422787 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790243735634835-388-0/minicluster-data/master-0/wal/instance
I20260924 09:55:44.423413 998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 09:55:44.423450 388 external_mini_cluster.cc:1428] Setting key 5a33e5a7703acdf673ae9a0c3b2216fd
I20260924 09:55:44.437216 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e74c3f5132fb454dadecb6e932fc3c5f: Bootstrap starting.
Sep 24 09:55:44 dist-test-slave-w8d2 krb5kdc[918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243743, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 09:55:44.441044 998 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e74c3f5132fb454dadecb6e932fc3c5f: Neither blocks nor log segments found. Creating new log.
I20260924 09:55:44.442332 998 log.cc:824] T 00000000000000000000000000000000 P e74c3f5132fb454dadecb6e932fc3c5f: Log is configured to *not* fsync() on all Append() calls
I20260924 09:55:44.445475 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e74c3f5132fb454dadecb6e932fc3c5f: No bootstrap required, opened a new log
I20260924 09:55:44.446465 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:55:44.425193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52374 (local address 127.0.97.62:40259)
0924 09:55:44.425905 (+ 712us) server_negotiation.cc:206] Beginning negotiation
0924 09:55:44.425916 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 09:55:44.425958 (+ 42us) server_negotiation.cc:407] Connection header received
0924 09:55:44.427772 (+ 1814us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:55:44.427792 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:55:44.428187 (+ 395us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:55:44.428519 (+ 332us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:55:44.430172 (+ 1653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:55:44.431477 (+ 1305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:55:44.433156 (+ 1679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:55:44.433886 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:55:44.434039 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:55:44.438442 (+ 4403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:55:44.438453 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:55:44.438473 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:55:44.438532 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:55:44.442166 (+ 3634us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:55:44.442798 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:55:44.442807 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:55:44.442817 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:55:44.443030 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:55:44.443426 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:55:44.443429 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:55:44.443431 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:55:44.444097 (+ 666us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:55:44.444355 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0924 09:55:44.444758 (+ 403us) server_negotiation.cc:299] Negotiation successful
0924 09:55:44.445163 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":508,"thread_start_us":302,"threads_started":1}
I20260924 09:55:44.452811 998 raft_consensus.cc:348] T 00000000000000000000000000000000 P e74c3f5132fb454dadecb6e932fc3c5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e74c3f5132fb454dadecb6e932fc3c5f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40259 } }
I20260924 09:55:44.453131 998 raft_consensus.cc:374] T 00000000000000000000000000000000 P e74c3f5132fb454dadecb6e932fc3c5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 09:55:44.453202 998 raft_consensus.cc:729] T 00000000000000000000000000000000 P e74c3f5132fb454dadecb6e932fc3c5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e74c3f5132fb454dadecb6e932fc3c5f, State: Initialized, Role: FOLLOWER
I20260924 09:55:44.453668 998 consensus_queue.cc:260] T 00000000000000000000000000000000 P e74c3f5132fb454dadecb6e932fc3c5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e74c3f5132fb454dadecb6e932fc3c5f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40259 } }
I20260924 09:55:44.453853 998 raft_consensus.cc:388] T 00000000000000000000000000000000 P e74c3f5132fb454dadecb6e932fc3c5f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 09:55:44.453939 998 raft_consensus.cc:482] T 00000000000000000000000000000000 P e74c3f5132fb454dadecb6e932fc3c5f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 09:55:44.454058 998 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e74c3f5132fb454dadecb6e932fc3c5f [term 0 FOLLOWER]: Advancing to term 1
I20260924 09:55:44.456071 998 raft_consensus.cc:504] T 00000000000000000000000000000000 P e74c3f5132fb454dadecb6e932fc3c5f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e74c3f5132fb454dadecb6e932fc3c5f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40259 } }
I20260924 09:55:44.456493 998 leader_election.cc:302] T 00000000000000000000000000000000 P e74c3f5132fb454dadecb6e932fc3c5f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e74c3f5132fb454dadecb6e932fc3c5f; no voters:
I20260924 09:55:44.457049 998 leader_election.cc:288] T 00000000000000000000000000000000 P e74c3f5132fb454dadecb6e932fc3c5f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 09:55:44.457170 1003 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e74c3f5132fb454dadecb6e932fc3c5f [term 1 FOLLOWER]: Leader election won for term 1
I20260924 09:55:44.457896 1003 raft_consensus.cc:686] T 00000000000000000000000000000000 P e74c3f5132fb454dadecb6e932fc3c5f [term 1 LEADER]: Becoming Leader. State: Replica: e74c3f5132fb454dadecb6e932fc3c5f, State: Running, Role: LEADER
I20260924 09:55:44.458441 1003 consensus_queue.cc:237] T 00000000000000000000000000000000 P e74c3f5132fb454dadecb6e932fc3c5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e74c3f5132fb454dadecb6e932fc3c5f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40259 } }
I20260924 09:55:44.459177 998 sys_catalog.cc:564] T 00000000000000000000000000000000 P e74c3f5132fb454dadecb6e932fc3c5f [sys.catalog]: configured and running, proceeding with master startup.
I20260924 09:55:44.461081 1005 sys_catalog.cc:455] T 00000000000000000000000000000000 P e74c3f5132fb454dadecb6e932fc3c5f [sys.catalog]: SysCatalogTable state changed. Reason: New leader e74c3f5132fb454dadecb6e932fc3c5f. Latest consensus state: current_term: 1 leader_uuid: "e74c3f5132fb454dadecb6e932fc3c5f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e74c3f5132fb454dadecb6e932fc3c5f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40259 } } }
I20260924 09:55:44.461547 1005 sys_catalog.cc:458] T 00000000000000000000000000000000 P e74c3f5132fb454dadecb6e932fc3c5f [sys.catalog]: This master's current role is: LEADER
I20260924 09:55:44.461098 1004 sys_catalog.cc:455] T 00000000000000000000000000000000 P e74c3f5132fb454dadecb6e932fc3c5f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e74c3f5132fb454dadecb6e932fc3c5f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e74c3f5132fb454dadecb6e932fc3c5f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40259 } } }
I20260924 09:55:44.462049 1004 sys_catalog.cc:458] T 00000000000000000000000000000000 P e74c3f5132fb454dadecb6e932fc3c5f [sys.catalog]: This master's current role is: LEADER
I20260924 09:55:44.463493 1008 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 09:55:44.468317 1008 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 09:55:44.476302 1008 catalog_manager.cc:1347] Generated new cluster ID: 094036edc7c542a9b1f8e506bf414a17
I20260924 09:55:44.476420 1008 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 09:55:44.496496 1008 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 09:55:44.497825 1008 catalog_manager.cc:1504] Loading token signing keys...
I20260924 09:55:44.510480 1008 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e74c3f5132fb454dadecb6e932fc3c5f: Generated new TSK 0
I20260924 09:55:44.511267 1008 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790243735634835-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790243735634835-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260924 09:55:44.556583 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
/tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790243735634835-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790243735634835-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790243735634835-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790243735634835-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40259
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790243735634835-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790243735634835-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790243735634835-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790243735634835-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 09:55:44.888478 1024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 09:55:44.888773 1024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 09:55:44.888880 1024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 09:55:44.895972 1024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 09:55:44.896071 1024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 09:55:44.896186 1024 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260924 09:55:44.905627 1024 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790243735634835-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790243735634835-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790243735634835-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790243735634835-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40259
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790243735634835-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 09:55:44.906777 1024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 09:55:44.908007 1024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 09:55:44.919891 1030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:55:44.919947 1032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:55:44.919972 1029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 09:55:44.920693 1024 server_base.cc:1034] running on GCE node
I20260924 09:55:44.921272 1024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 09:55:44.922156 1024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 09:55:44.923317 1024 hybrid_clock.cc:648] HybridClock initialized: now 1790243744923263 us; error 47 us; skew 500 ppm
I20260924 09:55:44.923599 1024 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 09:55:44 dist-test-slave-w8d2 krb5kdc[918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243744, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 09:55:44.926388 1024 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260924 09:55:44.927675 1024 webserver.cc:460] Webserver started at http://127.0.97.1:35453/ using document root <none> and password file <none>
I20260924 09:55:44.928287 1024 fs_manager.cc:359] Metadata directory not provided
I20260924 09:55:44.928387 1024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 09:55:44.928656 1024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 09:55:44.931361 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790243735634835-388-0/minicluster-data/ts-0/data/instance:
uuid: "3a88f350ab134eaeb41c2531acfb2f90"
format_stamp: "Formatted at 2026-09-24 09:55:44 on dist-test-slave-w8d2"
server_key: "499ea279dfe800042824b629ff2138fc"
server_key_iv: "663a2ecf2519797938791e633261d8df"
server_key_version: "encryptionkey@0"
I20260924 09:55:44.932108 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790243735634835-388-0/minicluster-data/ts-0/wal/instance:
uuid: "3a88f350ab134eaeb41c2531acfb2f90"
format_stamp: "Formatted at 2026-09-24 09:55:44 on dist-test-slave-w8d2"
server_key: "499ea279dfe800042824b629ff2138fc"
server_key_iv: "663a2ecf2519797938791e633261d8df"
server_key_version: "encryptionkey@0"
I20260924 09:55:44.936730 1024 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 09:55:44.939994 1039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 09:55:44.941265 1024 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 09:55:44.941417 1024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790243735634835-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790243735634835-388-0/minicluster-data/ts-0/wal
uuid: "3a88f350ab134eaeb41c2531acfb2f90"
format_stamp: "Formatted at 2026-09-24 09:55:44 on dist-test-slave-w8d2"
server_key: "499ea279dfe800042824b629ff2138fc"
server_key_iv: "663a2ecf2519797938791e633261d8df"
server_key_version: "encryptionkey@0"
I20260924 09:55:44.941591 1024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790243735634835-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790243735634835-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790243735634835-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 09:55:44.957988 1024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 09:55:44.961879 1024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 09:55:44.962138 1024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 09:55:44.963080 1024 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 09:55:44.964391 1024 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 09:55:44.964463 1024 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 09:55:44.964574 1024 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 09:55:44.964623 1024 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 09:55:44.992314 1024 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:36355
I20260924 09:55:44.992348 1151 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:36355 every 8 connection(s)
I20260924 09:55:44.993525 1024 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790243735634835-388-0/minicluster-data/ts-0/data/info.pb
I20260924 09:55:45.002959 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu as pid 1024
I20260924 09:55:45.003151 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790243735634835-388-0/minicluster-data/ts-0/wal/instance
I20260924 09:55:45.003875 388 external_mini_cluster.cc:1428] Setting key 63b48853f5c22a2e020e9c03d50b12d6
Sep 24 09:55:44 dist-test-slave-w8d2 krb5kdc[918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243744, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 09:55:45.011920 1154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:55:44.995982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:36279 (local address 127.0.97.62:40259)
0924 09:55:44.996376 (+ 394us) server_negotiation.cc:206] Beginning negotiation
0924 09:55:44.996380 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 09:55:44.997351 (+ 971us) server_negotiation.cc:407] Connection header received
0924 09:55:44.999022 (+ 1671us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:55:44.999026 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:55:44.999088 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:55:44.999180 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:55:45.000591 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:55:45.001293 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:55:45.002767 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:55:45.003354 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:55:45.003490 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:55:45.006751 (+ 3261us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:55:45.006756 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:55:45.006760 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:55:45.006789 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:55:45.009120 (+ 2331us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:55:45.009757 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:55:45.009761 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:55:45.009764 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:55:45.009873 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:55:45.010245 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:55:45.010250 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:55:45.010253 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:55:45.010523 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:55:45.010663 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0924 09:55:45.011355 (+ 692us) server_negotiation.cc:299] Negotiation successful
0924 09:55:45.011570 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":192,"threads_started":1}
I20260924 09:55:45.012851 1155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:55:44.996543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40259 (local address 127.0.97.1:36279)
0924 09:55:44.997206 (+ 663us) negotiation.cc:107] Waiting for socket to connect
0924 09:55:44.997242 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 09:55:44.998778 (+ 1536us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:55:44.999404 (+ 626us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:55:44.999413 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:55:44.999918 (+ 505us) client_negotiation.cc:189] Negotiated authn=SASL
0924 09:55:45.000440 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:55:45.000456 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:55:45.001419 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:55:45.001422 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:55:45.002630 (+ 1208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:55:45.002641 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:55:45.003561 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:55:45.003568 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:55:45.003780 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:55:45.004396 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 09:55:45.004417 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 09:55:45.006553 (+ 2136us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 09:55:45.009239 (+ 2686us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 09:55:45.009247 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 09:55:45.009265 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0924 09:55:45.009651 (+ 386us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 09:55:45.009976 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 09:55:45.009981 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 09:55:45.009985 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 09:55:45.010160 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 09:55:45.010636 (+ 476us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 09:55:45.010644 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 09:55:45.011007 (+ 363us) client_negotiation.cc:769] Sending connection context
0924 09:55:45.011261 (+ 254us) client_negotiation.cc:240] Negotiation successful
0924 09:55:45.011766 (+ 505us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":247,"threads_started":1}
I20260924 09:55:45.014302 1152 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40259
I20260924 09:55:45.014614 1152 heartbeater.cc:464] Registering TS with master...
I20260924 09:55:45.015488 1152 heartbeater.cc:511] Master 127.0.97.62:40259 requested a full tablet report, sending...
I20260924 09:55:45.017841 963 ts_manager.cc:194] Registered new tserver with Master: 3a88f350ab134eaeb41c2531acfb2f90 (127.0.97.1:36355)
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260924 09:55:45.019443 963 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:36279
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790243735634835-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790243735634835-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260924 09:55:45.041975 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
/tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790243735634835-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790243735634835-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790243735634835-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790243735634835-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40259
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790243735634835-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790243735634835-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790243735634835-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790243735634835-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 09:55:45.340827 1158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 09:55:45.341107 1158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 09:55:45.341205 1158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 09:55:45.348364 1158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 09:55:45.348471 1158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 09:55:45.348616 1158 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260924 09:55:45.358227 1158 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790243735634835-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790243735634835-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790243735634835-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790243735634835-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40259
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790243735634835-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 09:55:45.359526 1158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 09:55:45.360775 1158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 09:55:45.372920 1163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:55:45.372891 1166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:55:45.372854 1164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 09:55:45.373615 1158 server_base.cc:1034] running on GCE node
I20260924 09:55:45.374169 1158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 09:55:45.375053 1158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 09:55:45.376250 1158 hybrid_clock.cc:648] HybridClock initialized: now 1790243745376189 us; error 54 us; skew 500 ppm
I20260924 09:55:45.376569 1158 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 09:55:45 dist-test-slave-w8d2 krb5kdc[918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243745, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 09:55:45.379562 1158 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260924 09:55:45.380875 1158 webserver.cc:460] Webserver started at http://127.0.97.2:45671/ using document root <none> and password file <none>
I20260924 09:55:45.381539 1158 fs_manager.cc:359] Metadata directory not provided
I20260924 09:55:45.381635 1158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 09:55:45.381877 1158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 09:55:45.384677 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790243735634835-388-0/minicluster-data/ts-1/data/instance:
uuid: "75beed1b72464962a644475d7d29d9c5"
format_stamp: "Formatted at 2026-09-24 09:55:45 on dist-test-slave-w8d2"
server_key: "827aa4fefca5e3ee1267f4a4953815b6"
server_key_iv: "a1587e5b79268a62789f9869e811c27c"
server_key_version: "encryptionkey@0"
I20260924 09:55:45.385389 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790243735634835-388-0/minicluster-data/ts-1/wal/instance:
uuid: "75beed1b72464962a644475d7d29d9c5"
format_stamp: "Formatted at 2026-09-24 09:55:45 on dist-test-slave-w8d2"
server_key: "827aa4fefca5e3ee1267f4a4953815b6"
server_key_iv: "a1587e5b79268a62789f9869e811c27c"
server_key_version: "encryptionkey@0"
I20260924 09:55:45.390467 1158 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260924 09:55:45.393654 1173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 09:55:45.394906 1158 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260924 09:55:45.395032 1158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790243735634835-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790243735634835-388-0/minicluster-data/ts-1/wal
uuid: "75beed1b72464962a644475d7d29d9c5"
format_stamp: "Formatted at 2026-09-24 09:55:45 on dist-test-slave-w8d2"
server_key: "827aa4fefca5e3ee1267f4a4953815b6"
server_key_iv: "a1587e5b79268a62789f9869e811c27c"
server_key_version: "encryptionkey@0"
I20260924 09:55:45.395195 1158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790243735634835-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790243735634835-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790243735634835-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 09:55:45.410077 1158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 09:55:45.413676 1158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 09:55:45.413905 1158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 09:55:45.414734 1158 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 09:55:45.416013 1158 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 09:55:45.416082 1158 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 09:55:45.416189 1158 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 09:55:45.416239 1158 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 09:55:45.445104 1158 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:41117
I20260924 09:55:45.445145 1285 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:41117 every 8 connection(s)
I20260924 09:55:45.446376 1158 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790243735634835-388-0/minicluster-data/ts-1/data/info.pb
I20260924 09:55:45.455818 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu as pid 1158
I20260924 09:55:45.455962 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790243735634835-388-0/minicluster-data/ts-1/wal/instance
I20260924 09:55:45.456514 388 external_mini_cluster.cc:1428] Setting key a8508ed4d68fc9c4384dde8ebf123f9c
Sep 24 09:55:45 dist-test-slave-w8d2 krb5kdc[918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243745, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 09:55:45.465190 1154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:55:45.448598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:58475 (local address 127.0.97.62:40259)
0924 09:55:45.448798 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0924 09:55:45.448803 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 09:55:45.449926 (+ 1123us) server_negotiation.cc:407] Connection header received
0924 09:55:45.451690 (+ 1764us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:55:45.451694 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:55:45.451759 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:55:45.451855 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:55:45.453186 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:55:45.453926 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:55:45.455354 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:55:45.455873 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:55:45.456028 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:55:45.459803 (+ 3775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:55:45.459807 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:55:45.459811 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:55:45.459840 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:55:45.462246 (+ 2406us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:55:45.462935 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:55:45.462941 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:55:45.462944 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:55:45.463099 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:55:45.463569 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:55:45.463573 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:55:45.463576 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:55:45.463877 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:55:45.464022 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0924 09:55:45.464642 (+ 620us) server_negotiation.cc:299] Negotiation successful
0924 09:55:45.464861 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260924 09:55:45.465950 1288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:55:45.449108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40259 (local address 127.0.97.2:58475)
0924 09:55:45.449760 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0924 09:55:45.449813 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0924 09:55:45.451366 (+ 1553us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:55:45.452032 (+ 666us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:55:45.452039 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:55:45.452503 (+ 464us) client_negotiation.cc:189] Negotiated authn=SASL
0924 09:55:45.453026 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:55:45.453041 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:55:45.454038 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:55:45.454041 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:55:45.455209 (+ 1168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:55:45.455219 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:55:45.456133 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:55:45.456140 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:55:45.456415 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:55:45.457060 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 09:55:45.457085 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0924 09:55:45.459614 (+ 2529us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 09:55:45.462378 (+ 2764us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 09:55:45.462387 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 09:55:45.462405 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0924 09:55:45.462817 (+ 412us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 09:55:45.463208 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 09:55:45.463211 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 09:55:45.463213 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 09:55:45.463419 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 09:55:45.463978 (+ 559us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 09:55:45.463983 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 09:55:45.464289 (+ 306us) client_negotiation.cc:769] Sending connection context
0924 09:55:45.464522 (+ 233us) client_negotiation.cc:240] Negotiation successful
0924 09:55:45.464892 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":248,"threads_started":1}
I20260924 09:55:45.467504 1286 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40259
I20260924 09:55:45.467809 1286 heartbeater.cc:464] Registering TS with master...
I20260924 09:55:45.468544 1286 heartbeater.cc:511] Master 127.0.97.62:40259 requested a full tablet report, sending...
I20260924 09:55:45.469961 963 ts_manager.cc:194] Registered new tserver with Master: 75beed1b72464962a644475d7d29d9c5 (127.0.97.2:41117)
I20260924 09:55:45.471048 963 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:58475
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790243735634835-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790243735634835-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260924 09:55:45.492213 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
/tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790243735634835-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790243735634835-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790243735634835-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790243735634835-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40259
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790243735634835-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790243735634835-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790243735634835-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790243735634835-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 09:55:45.796993 1291 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 09:55:45.797273 1291 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 09:55:45.797374 1291 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 09:55:45.804459 1291 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 09:55:45.804569 1291 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 09:55:45.804688 1291 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260924 09:55:45.814124 1291 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790243735634835-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790243735634835-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790243735634835-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790243735634835-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40259
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790243735634835-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 09:55:45.815435 1291 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 09:55:45.816663 1291 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 09:55:45.828641 1297 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:55:45.828599 1296 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:55:45.828675 1299 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 09:55:45.829391 1291 server_base.cc:1034] running on GCE node
I20260924 09:55:45.829949 1291 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 09:55:45.830794 1291 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 09:55:45.831979 1291 hybrid_clock.cc:648] HybridClock initialized: now 1790243745831928 us; error 39 us; skew 500 ppm
I20260924 09:55:45.832260 1291 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 09:55:45 dist-test-slave-w8d2 krb5kdc[918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243745, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 09:55:45.834946 1291 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260924 09:55:45.836310 1291 webserver.cc:460] Webserver started at http://127.0.97.3:37593/ using document root <none> and password file <none>
I20260924 09:55:45.836928 1291 fs_manager.cc:359] Metadata directory not provided
I20260924 09:55:45.837036 1291 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 09:55:45.837316 1291 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 09:55:45.840057 1291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790243735634835-388-0/minicluster-data/ts-2/data/instance:
uuid: "7b1317f6183249448df0505cf33da248"
format_stamp: "Formatted at 2026-09-24 09:55:45 on dist-test-slave-w8d2"
server_key: "00783a5716d212c9444e03a5b21c4007"
server_key_iv: "835e0f9e06409f8205efb67413308ec3"
server_key_version: "encryptionkey@0"
I20260924 09:55:45.840778 1291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790243735634835-388-0/minicluster-data/ts-2/wal/instance:
uuid: "7b1317f6183249448df0505cf33da248"
format_stamp: "Formatted at 2026-09-24 09:55:45 on dist-test-slave-w8d2"
server_key: "00783a5716d212c9444e03a5b21c4007"
server_key_iv: "835e0f9e06409f8205efb67413308ec3"
server_key_version: "encryptionkey@0"
I20260924 09:55:45.845614 1291 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260924 09:55:45.848738 1306 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 09:55:45.849941 1291 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 09:55:45.850087 1291 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790243735634835-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790243735634835-388-0/minicluster-data/ts-2/wal
uuid: "7b1317f6183249448df0505cf33da248"
format_stamp: "Formatted at 2026-09-24 09:55:45 on dist-test-slave-w8d2"
server_key: "00783a5716d212c9444e03a5b21c4007"
server_key_iv: "835e0f9e06409f8205efb67413308ec3"
server_key_version: "encryptionkey@0"
I20260924 09:55:45.850253 1291 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790243735634835-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790243735634835-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790243735634835-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 09:55:45.872463 1291 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 09:55:45.876276 1291 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 09:55:45.876528 1291 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 09:55:45.877434 1291 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 09:55:45.878651 1291 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 09:55:45.878726 1291 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 09:55:45.878839 1291 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 09:55:45.878890 1291 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 09:55:45.906834 1291 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:38193
I20260924 09:55:45.906872 1418 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:38193 every 8 connection(s)
I20260924 09:55:45.908066 1291 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790243735634835-388-0/minicluster-data/ts-2/data/info.pb
I20260924 09:55:45.916764 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu as pid 1291
I20260924 09:55:45.916905 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790243735634835-388-0/minicluster-data/ts-2/wal/instance
I20260924 09:55:45.917413 388 external_mini_cluster.cc:1428] Setting key 2a52107d3cf838e36e64298f98366a2d
Sep 24 09:55:45 dist-test-slave-w8d2 krb5kdc[918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243745, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 09:55:45.925694 1154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:55:45.910309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:42221 (local address 127.0.97.62:40259)
0924 09:55:45.910466 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0924 09:55:45.910470 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 09:55:45.911587 (+ 1117us) server_negotiation.cc:407] Connection header received
0924 09:55:45.913252 (+ 1665us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:55:45.913256 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:55:45.913319 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:55:45.913399 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:55:45.914729 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:55:45.915479 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:55:45.916910 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:55:45.917496 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:55:45.917618 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:55:45.921049 (+ 3431us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:55:45.921054 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:55:45.921057 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:55:45.921096 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:55:45.923418 (+ 2322us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:55:45.924005 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:55:45.924008 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:55:45.924011 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:55:45.924108 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:55:45.924413 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:55:45.924415 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:55:45.924417 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:55:45.924650 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:55:45.924750 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0924 09:55:45.925307 (+ 557us) server_negotiation.cc:299] Negotiation successful
0924 09:55:45.925452 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 09:55:45.926669 1421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:55:45.910766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40259 (local address 127.0.97.3:42221)
0924 09:55:45.911438 (+ 672us) negotiation.cc:107] Waiting for socket to connect
0924 09:55:45.911480 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 09:55:45.913018 (+ 1538us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:55:45.913575 (+ 557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:55:45.913582 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:55:45.914056 (+ 474us) client_negotiation.cc:189] Negotiated authn=SASL
0924 09:55:45.914593 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:55:45.914609 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:55:45.915590 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:55:45.915594 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:55:45.916770 (+ 1176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:55:45.916781 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:55:45.917685 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:55:45.917690 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:55:45.917871 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:55:45.918569 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 09:55:45.918596 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0924 09:55:45.920883 (+ 2287us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 09:55:45.923516 (+ 2633us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 09:55:45.923522 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 09:55:45.923534 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0924 09:55:45.923925 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 09:55:45.924181 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 09:55:45.924183 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 09:55:45.924186 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 09:55:45.924342 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 09:55:45.924735 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 09:55:45.924741 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 09:55:45.925027 (+ 286us) client_negotiation.cc:769] Sending connection context
0924 09:55:45.925237 (+ 210us) client_negotiation.cc:240] Negotiation successful
0924 09:55:45.925602 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":246,"threads_started":1}
I20260924 09:55:45.927909 1419 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40259
I20260924 09:55:45.928180 1419 heartbeater.cc:464] Registering TS with master...
I20260924 09:55:45.928798 1419 heartbeater.cc:511] Master 127.0.97.62:40259 requested a full tablet report, sending...
I20260924 09:55:45.930069 963 ts_manager.cc:194] Registered new tserver with Master: 7b1317f6183249448df0505cf33da248 (127.0.97.3:38193)
I20260924 09:55:45.931401 963 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:42221
I20260924 09:55:45.932453 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 09:55:45.949064 1154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:55:45.938022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52384 (local address 127.0.97.62:40259)
0924 09:55:45.938189 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0924 09:55:45.938194 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 09:55:45.938623 (+ 429us) server_negotiation.cc:407] Connection header received
0924 09:55:45.938755 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:55:45.938758 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:55:45.938813 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:55:45.938890 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:55:45.940272 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:55:45.940930 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:55:45.942115 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:55:45.942487 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:55:45.942569 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:55:45.944581 (+ 2012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:55:45.944585 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:55:45.944588 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:55:45.944616 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:55:45.946960 (+ 2344us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:55:45.947574 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:55:45.947580 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:55:45.947583 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:55:45.947691 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:55:45.948010 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:55:45.948013 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:55:45.948015 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:55:45.948285 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:55:45.948461 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0924 09:55:45.948705 (+ 244us) server_negotiation.cc:299] Negotiation successful
0924 09:55:45.948853 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 09:55:45.955192 963 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52384:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 09:55:45.961835 963 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 09:55:45.983541 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:55:45.973047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39196 (local address 127.0.97.1:36355)
0924 09:55:45.974844 (+ 1797us) server_negotiation.cc:206] Beginning negotiation
0924 09:55:45.974853 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 09:55:45.974879 (+ 26us) server_negotiation.cc:407] Connection header received
0924 09:55:45.975347 (+ 468us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:55:45.975355 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:55:45.975611 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:55:45.976441 (+ 830us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 09:55:45.976638 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:55:45.978070 (+ 1432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:55:45.980587 (+ 2517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:55:45.982523 (+ 1936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:55:45.982652 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:55:45.982846 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0924 09:55:45.983126 (+ 280us) server_negotiation.cc:299] Negotiation successful
0924 09:55:45.983307 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1667,"thread_start_us":1577,"threads_started":1}
I20260924 09:55:45.983623 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:55:45.972854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36355 (local address 127.0.0.1:39196)
0924 09:55:45.973702 (+ 848us) negotiation.cc:107] Waiting for socket to connect
0924 09:55:45.973749 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0924 09:55:45.975061 (+ 1312us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:55:45.975834 (+ 773us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:55:45.975841 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:55:45.975871 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 09:55:45.976252 (+ 381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:55:45.976265 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:55:45.978215 (+ 1950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:55:45.978219 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:55:45.980447 (+ 2228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:55:45.980458 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:55:45.982694 (+ 2236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:55:45.982700 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:55:45.982893 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:55:45.982944 (+ 51us) client_negotiation.cc:769] Sending connection context
0924 09:55:45.983087 (+ 143us) client_negotiation.cc:240] Negotiation successful
0924 09:55:45.983292 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":586,"thread_start_us":3930,"threads_started":1}
I20260924 09:55:45.984612 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:55:45.974751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51026 (local address 127.0.97.3:38193)
0924 09:55:45.976395 (+ 1644us) server_negotiation.cc:206] Beginning negotiation
0924 09:55:45.976403 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 09:55:45.977146 (+ 743us) server_negotiation.cc:407] Connection header received
0924 09:55:45.977256 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:55:45.977263 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:55:45.977463 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:55:45.977645 (+ 182us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 09:55:45.978248 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:55:45.979558 (+ 1310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:55:45.981685 (+ 2127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:55:45.982986 (+ 1301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:55:45.983110 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:55:45.983354 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0924 09:55:45.984154 (+ 800us) server_negotiation.cc:299] Negotiation successful
0924 09:55:45.984312 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1485,"thread_start_us":1352,"threads_started":1}
I20260924 09:55:45.984629 1431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:55:45.974048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:38193 (local address 127.0.0.1:51026)
0924 09:55:45.977047 (+ 2999us) negotiation.cc:107] Waiting for socket to connect
0924 09:55:45.977060 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 09:55:45.977148 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:55:45.977663 (+ 515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:55:45.977667 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:55:45.977697 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 09:55:45.978060 (+ 363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:55:45.978073 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:55:45.979669 (+ 1596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:55:45.979674 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:55:45.981424 (+ 1750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:55:45.981443 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:55:45.983837 (+ 2394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:55:45.983843 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:55:45.983996 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:55:45.984026 (+ 30us) client_negotiation.cc:769] Sending connection context
0924 09:55:45.984101 (+ 75us) client_negotiation.cc:240] Negotiation successful
0924 09:55:45.984218 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2880,"mutex_wait_us":64,"thread_start_us":2487,"threads_started":1}
I20260924 09:55:45.985116 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:55:45.972989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:41117 (local address 127.0.0.1:36814)
0924 09:55:45.976800 (+ 3811us) negotiation.cc:107] Waiting for socket to connect
0924 09:55:45.976817 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 09:55:45.976935 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:55:45.978373 (+ 1438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:55:45.978378 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:55:45.978404 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 09:55:45.978791 (+ 387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:55:45.978802 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:55:45.980610 (+ 1808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:55:45.980615 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:55:45.982347 (+ 1732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:55:45.982358 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:55:45.984357 (+ 1999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:55:45.984363 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:55:45.984519 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:55:45.984551 (+ 32us) client_negotiation.cc:769] Sending connection context
0924 09:55:45.984654 (+ 103us) client_negotiation.cc:240] Negotiation successful
0924 09:55:45.984766 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3687,"mutex_wait_us":54,"thread_start_us":331,"threads_started":1}
I20260924 09:55:45.985339 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:55:45.973512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36814 (local address 127.0.97.2:41117)
0924 09:55:45.977394 (+ 3882us) server_negotiation.cc:206] Beginning negotiation
0924 09:55:45.977400 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 09:55:45.977419 (+ 19us) server_negotiation.cc:407] Connection header received
0924 09:55:45.977490 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:55:45.977494 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:55:45.977966 (+ 472us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:55:45.978143 (+ 177us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 09:55:45.978974 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:55:45.980444 (+ 1470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:55:45.982486 (+ 2042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:55:45.983941 (+ 1455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:55:45.984052 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:55:45.984219 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0924 09:55:45.984891 (+ 672us) server_negotiation.cc:299] Negotiation successful
0924 09:55:45.985053 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3772,"thread_start_us":1471,"threads_started":1}
I20260924 09:55:45.987516 1087 tablet_service.cc:1467] Processing CreateTablet for tablet 191b15005290429fb630741e7663c502 (DEFAULT_TABLE table=test-table [id=b98d40af22714c65b94db8c7f90f95ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 09:55:45.988731 1087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 191b15005290429fb630741e7663c502. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 09:55:45.988663 1221 tablet_service.cc:1467] Processing CreateTablet for tablet 191b15005290429fb630741e7663c502 (DEFAULT_TABLE table=test-table [id=b98d40af22714c65b94db8c7f90f95ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 09:55:45.989979 1221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 191b15005290429fb630741e7663c502. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 09:55:45.989845 1354 tablet_service.cc:1467] Processing CreateTablet for tablet 191b15005290429fb630741e7663c502 (DEFAULT_TABLE table=test-table [id=b98d40af22714c65b94db8c7f90f95ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 09:55:45.991150 1354 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 191b15005290429fb630741e7663c502. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 09:55:46.005331 1439 tablet_bootstrap.cc:492] T 191b15005290429fb630741e7663c502 P 75beed1b72464962a644475d7d29d9c5: Bootstrap starting.
I20260924 09:55:46.007197 1438 tablet_bootstrap.cc:492] T 191b15005290429fb630741e7663c502 P 7b1317f6183249448df0505cf33da248: Bootstrap starting.
I20260924 09:55:46.008319 1437 tablet_bootstrap.cc:492] T 191b15005290429fb630741e7663c502 P 3a88f350ab134eaeb41c2531acfb2f90: Bootstrap starting.
I20260924 09:55:46.008950 1439 tablet_bootstrap.cc:654] T 191b15005290429fb630741e7663c502 P 75beed1b72464962a644475d7d29d9c5: Neither blocks nor log segments found. Creating new log.
I20260924 09:55:46.009852 1439 log.cc:824] T 191b15005290429fb630741e7663c502 P 75beed1b72464962a644475d7d29d9c5: Log is configured to *not* fsync() on all Append() calls
I20260924 09:55:46.011075 1438 tablet_bootstrap.cc:654] T 191b15005290429fb630741e7663c502 P 7b1317f6183249448df0505cf33da248: Neither blocks nor log segments found. Creating new log.
I20260924 09:55:46.011472 1437 tablet_bootstrap.cc:654] T 191b15005290429fb630741e7663c502 P 3a88f350ab134eaeb41c2531acfb2f90: Neither blocks nor log segments found. Creating new log.
I20260924 09:55:46.012204 1438 log.cc:824] T 191b15005290429fb630741e7663c502 P 7b1317f6183249448df0505cf33da248: Log is configured to *not* fsync() on all Append() calls
I20260924 09:55:46.012336 1439 tablet_bootstrap.cc:492] T 191b15005290429fb630741e7663c502 P 75beed1b72464962a644475d7d29d9c5: No bootstrap required, opened a new log
I20260924 09:55:46.012485 1437 log.cc:824] T 191b15005290429fb630741e7663c502 P 3a88f350ab134eaeb41c2531acfb2f90: Log is configured to *not* fsync() on all Append() calls
I20260924 09:55:46.012616 1439 ts_tablet_manager.cc:1397] T 191b15005290429fb630741e7663c502 P 75beed1b72464962a644475d7d29d9c5: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.005s
I20260924 09:55:46.014881 1437 tablet_bootstrap.cc:492] T 191b15005290429fb630741e7663c502 P 3a88f350ab134eaeb41c2531acfb2f90: No bootstrap required, opened a new log
I20260924 09:55:46.015097 1437 ts_tablet_manager.cc:1397] T 191b15005290429fb630741e7663c502 P 3a88f350ab134eaeb41c2531acfb2f90: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260924 09:55:46.015504 1438 tablet_bootstrap.cc:492] T 191b15005290429fb630741e7663c502 P 7b1317f6183249448df0505cf33da248: No bootstrap required, opened a new log
I20260924 09:55:46.015728 1438 ts_tablet_manager.cc:1397] T 191b15005290429fb630741e7663c502 P 7b1317f6183249448df0505cf33da248: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.003s
I20260924 09:55:46.021840 1437 raft_consensus.cc:348] T 191b15005290429fb630741e7663c502 P 3a88f350ab134eaeb41c2531acfb2f90 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a88f350ab134eaeb41c2531acfb2f90" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36355 } } peers { permanent_uuid: "75beed1b72464962a644475d7d29d9c5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41117 } } peers { permanent_uuid: "7b1317f6183249448df0505cf33da248" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38193 } }
I20260924 09:55:46.022228 1152 heartbeater.cc:503] Master 127.0.97.62:40259 was elected leader, sending a full tablet report...
I20260924 09:55:46.022257 1437 raft_consensus.cc:374] T 191b15005290429fb630741e7663c502 P 3a88f350ab134eaeb41c2531acfb2f90 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 09:55:46.022377 1437 raft_consensus.cc:729] T 191b15005290429fb630741e7663c502 P 3a88f350ab134eaeb41c2531acfb2f90 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a88f350ab134eaeb41c2531acfb2f90, State: Initialized, Role: FOLLOWER
I20260924 09:55:46.023015 1437 consensus_queue.cc:260] T 191b15005290429fb630741e7663c502 P 3a88f350ab134eaeb41c2531acfb2f90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a88f350ab134eaeb41c2531acfb2f90" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36355 } } peers { permanent_uuid: "75beed1b72464962a644475d7d29d9c5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41117 } } peers { permanent_uuid: "7b1317f6183249448df0505cf33da248" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38193 } }
I20260924 09:55:46.022728 1439 raft_consensus.cc:348] T 191b15005290429fb630741e7663c502 P 75beed1b72464962a644475d7d29d9c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a88f350ab134eaeb41c2531acfb2f90" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36355 } } peers { permanent_uuid: "75beed1b72464962a644475d7d29d9c5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41117 } } peers { permanent_uuid: "7b1317f6183249448df0505cf33da248" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38193 } }
I20260924 09:55:46.023847 1439 raft_consensus.cc:374] T 191b15005290429fb630741e7663c502 P 75beed1b72464962a644475d7d29d9c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 09:55:46.023934 1439 raft_consensus.cc:729] T 191b15005290429fb630741e7663c502 P 75beed1b72464962a644475d7d29d9c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75beed1b72464962a644475d7d29d9c5, State: Initialized, Role: FOLLOWER
I20260924 09:55:46.024400 1439 consensus_queue.cc:260] T 191b15005290429fb630741e7663c502 P 75beed1b72464962a644475d7d29d9c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a88f350ab134eaeb41c2531acfb2f90" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36355 } } peers { permanent_uuid: "75beed1b72464962a644475d7d29d9c5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41117 } } peers { permanent_uuid: "7b1317f6183249448df0505cf33da248" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38193 } }
I20260924 09:55:46.025327 1286 heartbeater.cc:503] Master 127.0.97.62:40259 was elected leader, sending a full tablet report...
I20260924 09:55:46.026140 1438 raft_consensus.cc:348] T 191b15005290429fb630741e7663c502 P 7b1317f6183249448df0505cf33da248 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a88f350ab134eaeb41c2531acfb2f90" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36355 } } peers { permanent_uuid: "75beed1b72464962a644475d7d29d9c5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41117 } } peers { permanent_uuid: "7b1317f6183249448df0505cf33da248" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38193 } }
I20260924 09:55:46.026539 1438 raft_consensus.cc:374] T 191b15005290429fb630741e7663c502 P 7b1317f6183249448df0505cf33da248 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 09:55:46.026655 1438 raft_consensus.cc:729] T 191b15005290429fb630741e7663c502 P 7b1317f6183249448df0505cf33da248 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b1317f6183249448df0505cf33da248, State: Initialized, Role: FOLLOWER
I20260924 09:55:46.026662 1439 ts_tablet_manager.cc:1428] T 191b15005290429fb630741e7663c502 P 75beed1b72464962a644475d7d29d9c5: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260924 09:55:46.027248 1438 consensus_queue.cc:260] T 191b15005290429fb630741e7663c502 P 7b1317f6183249448df0505cf33da248 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a88f350ab134eaeb41c2531acfb2f90" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36355 } } peers { permanent_uuid: "75beed1b72464962a644475d7d29d9c5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41117 } } peers { permanent_uuid: "7b1317f6183249448df0505cf33da248" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38193 } }
I20260924 09:55:46.028447 1419 heartbeater.cc:503] Master 127.0.97.62:40259 was elected leader, sending a full tablet report...
I20260924 09:55:46.028923 1438 ts_tablet_manager.cc:1428] T 191b15005290429fb630741e7663c502 P 7b1317f6183249448df0505cf33da248: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
I20260924 09:55:46.032496 1437 ts_tablet_manager.cc:1428] T 191b15005290429fb630741e7663c502 P 3a88f350ab134eaeb41c2531acfb2f90: Time spent starting tablet: real 0.017s user 0.007s sys 0.005s
W20260924 09:55:46.159452 1420 tablet.cc:2369] T 191b15005290429fb630741e7663c502 P 7b1317f6183249448df0505cf33da248: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 09:55:46.190986 1443 raft_consensus.cc:482] T 191b15005290429fb630741e7663c502 P 75beed1b72464962a644475d7d29d9c5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 09:55:46.191300 1443 raft_consensus.cc:504] T 191b15005290429fb630741e7663c502 P 75beed1b72464962a644475d7d29d9c5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a88f350ab134eaeb41c2531acfb2f90" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36355 } } peers { permanent_uuid: "75beed1b72464962a644475d7d29d9c5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41117 } } peers { permanent_uuid: "7b1317f6183249448df0505cf33da248" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38193 } }
I20260924 09:55:46.192874 1443 leader_election.cc:288] T 191b15005290429fb630741e7663c502 P 75beed1b72464962a644475d7d29d9c5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3a88f350ab134eaeb41c2531acfb2f90 (127.0.97.1:36355), 7b1317f6183249448df0505cf33da248 (127.0.97.3:38193)
W20260924 09:55:46.198167 1287 tablet.cc:2369] T 191b15005290429fb630741e7663c502 P 75beed1b72464962a644475d7d29d9c5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 09:55:46.199270 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:55:46.193364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:43583 (local address 127.0.97.1:36355)
0924 09:55:46.193554 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0924 09:55:46.193559 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 09:55:46.194067 (+ 508us) server_negotiation.cc:407] Connection header received
0924 09:55:46.194247 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:55:46.194251 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:55:46.194309 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:55:46.194379 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 09:55:46.194909 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:55:46.195629 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:55:46.197446 (+ 1817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:55:46.198393 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:55:46.198517 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:55:46.198600 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0924 09:55:46.198882 (+ 282us) server_negotiation.cc:299] Negotiation successful
0924 09:55:46.198995 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260924 09:55:46.199661 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:55:46.193240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:55165 (local address 127.0.97.3:38193)
0924 09:55:46.193427 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0924 09:55:46.193433 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 09:55:46.194223 (+ 790us) server_negotiation.cc:407] Connection header received
0924 09:55:46.194371 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:55:46.194375 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:55:46.194429 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:55:46.194502 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 09:55:46.195061 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:55:46.195790 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:55:46.197425 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:55:46.198197 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:55:46.198308 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:55:46.198423 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0924 09:55:46.198857 (+ 434us) server_negotiation.cc:299] Negotiation successful
0924 09:55:46.198983 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 09:55:46.199739 1447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:55:46.193405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:38193 (local address 127.0.97.2:55165)
0924 09:55:46.194119 (+ 714us) negotiation.cc:107] Waiting for socket to connect
0924 09:55:46.194132 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 09:55:46.194253 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:55:46.194527 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:55:46.194531 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:55:46.194568 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 09:55:46.194932 (+ 364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:55:46.194940 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:55:46.195914 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:55:46.195919 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:55:46.197296 (+ 1377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:55:46.197307 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:55:46.198398 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:55:46.198404 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:55:46.198635 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:55:46.198668 (+ 33us) client_negotiation.cc:769] Sending connection context
0924 09:55:46.198804 (+ 136us) client_negotiation.cc:240] Negotiation successful
0924 09:55:46.198921 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":622,"mutex_wait_us":1,"thread_start_us":437,"threads_started":1}
I20260924 09:55:46.200075 1446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:55:46.193296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36355 (local address 127.0.97.2:43583)
0924 09:55:46.193962 (+ 666us) negotiation.cc:107] Waiting for socket to connect
0924 09:55:46.193977 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 09:55:46.194120 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:55:46.194403 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:55:46.194407 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:55:46.194429 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 09:55:46.194772 (+ 343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:55:46.194781 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:55:46.195748 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:55:46.195753 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:55:46.197307 (+ 1554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:55:46.197317 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:55:46.198590 (+ 1273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:55:46.198596 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:55:46.198729 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:55:46.198756 (+ 27us) client_negotiation.cc:769] Sending connection context
0924 09:55:46.199431 (+ 675us) client_negotiation.cc:240] Negotiation successful
0924 09:55:46.199556 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":532,"mutex_wait_us":72,"spinlock_wait_cycles":17152,"thread_start_us":325,"threads_started":1}
I20260924 09:55:46.200568 1374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "191b15005290429fb630741e7663c502" candidate_uuid: "75beed1b72464962a644475d7d29d9c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b1317f6183249448df0505cf33da248" is_pre_election: true
I20260924 09:55:46.200848 1107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "191b15005290429fb630741e7663c502" candidate_uuid: "75beed1b72464962a644475d7d29d9c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a88f350ab134eaeb41c2531acfb2f90" is_pre_election: true
I20260924 09:55:46.200984 1374 raft_consensus.cc:2454] T 191b15005290429fb630741e7663c502 P 7b1317f6183249448df0505cf33da248 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 75beed1b72464962a644475d7d29d9c5 in term 0.
I20260924 09:55:46.201275 1107 raft_consensus.cc:2454] T 191b15005290429fb630741e7663c502 P 3a88f350ab134eaeb41c2531acfb2f90 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 75beed1b72464962a644475d7d29d9c5 in term 0.
I20260924 09:55:46.201568 1177 leader_election.cc:302] T 191b15005290429fb630741e7663c502 P 75beed1b72464962a644475d7d29d9c5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 75beed1b72464962a644475d7d29d9c5, 7b1317f6183249448df0505cf33da248; no voters:
I20260924 09:55:46.201869 1443 raft_consensus.cc:2781] T 191b15005290429fb630741e7663c502 P 75beed1b72464962a644475d7d29d9c5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 09:55:46.201978 1443 raft_consensus.cc:482] T 191b15005290429fb630741e7663c502 P 75beed1b72464962a644475d7d29d9c5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 09:55:46.202070 1443 raft_consensus.cc:3037] T 191b15005290429fb630741e7663c502 P 75beed1b72464962a644475d7d29d9c5 [term 0 FOLLOWER]: Advancing to term 1
I20260924 09:55:46.204176 1443 raft_consensus.cc:504] T 191b15005290429fb630741e7663c502 P 75beed1b72464962a644475d7d29d9c5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a88f350ab134eaeb41c2531acfb2f90" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36355 } } peers { permanent_uuid: "75beed1b72464962a644475d7d29d9c5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41117 } } peers { permanent_uuid: "7b1317f6183249448df0505cf33da248" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38193 } }
I20260924 09:55:46.204712 1443 leader_election.cc:288] T 191b15005290429fb630741e7663c502 P 75beed1b72464962a644475d7d29d9c5 [CANDIDATE]: Term 1 election: Requested vote from peers 3a88f350ab134eaeb41c2531acfb2f90 (127.0.97.1:36355), 7b1317f6183249448df0505cf33da248 (127.0.97.3:38193)
I20260924 09:55:46.205062 1107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "191b15005290429fb630741e7663c502" candidate_uuid: "75beed1b72464962a644475d7d29d9c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a88f350ab134eaeb41c2531acfb2f90"
I20260924 09:55:46.205099 1374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "191b15005290429fb630741e7663c502" candidate_uuid: "75beed1b72464962a644475d7d29d9c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b1317f6183249448df0505cf33da248"
I20260924 09:55:46.205228 1107 raft_consensus.cc:3037] T 191b15005290429fb630741e7663c502 P 3a88f350ab134eaeb41c2531acfb2f90 [term 0 FOLLOWER]: Advancing to term 1
I20260924 09:55:46.205283 1374 raft_consensus.cc:3037] T 191b15005290429fb630741e7663c502 P 7b1317f6183249448df0505cf33da248 [term 0 FOLLOWER]: Advancing to term 1
I20260924 09:55:46.207531 1374 raft_consensus.cc:2454] T 191b15005290429fb630741e7663c502 P 7b1317f6183249448df0505cf33da248 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 75beed1b72464962a644475d7d29d9c5 in term 1.
I20260924 09:55:46.207538 1107 raft_consensus.cc:2454] T 191b15005290429fb630741e7663c502 P 3a88f350ab134eaeb41c2531acfb2f90 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 75beed1b72464962a644475d7d29d9c5 in term 1.
I20260924 09:55:46.207904 1174 leader_election.cc:302] T 191b15005290429fb630741e7663c502 P 75beed1b72464962a644475d7d29d9c5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a88f350ab134eaeb41c2531acfb2f90, 75beed1b72464962a644475d7d29d9c5; no voters:
I20260924 09:55:46.208213 1443 raft_consensus.cc:2781] T 191b15005290429fb630741e7663c502 P 75beed1b72464962a644475d7d29d9c5 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 09:55:46.208575 1443 raft_consensus.cc:686] T 191b15005290429fb630741e7663c502 P 75beed1b72464962a644475d7d29d9c5 [term 1 LEADER]: Becoming Leader. State: Replica: 75beed1b72464962a644475d7d29d9c5, State: Running, Role: LEADER
I20260924 09:55:46.209168 1443 consensus_queue.cc:237] T 191b15005290429fb630741e7663c502 P 75beed1b72464962a644475d7d29d9c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a88f350ab134eaeb41c2531acfb2f90" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36355 } } peers { permanent_uuid: "75beed1b72464962a644475d7d29d9c5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41117 } } peers { permanent_uuid: "7b1317f6183249448df0505cf33da248" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38193 } }
I20260924 09:55:46.213214 962 catalog_manager.cc:5495] T 191b15005290429fb630741e7663c502 P 75beed1b72464962a644475d7d29d9c5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 75beed1b72464962a644475d7d29d9c5 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "75beed1b72464962a644475d7d29d9c5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a88f350ab134eaeb41c2531acfb2f90" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "75beed1b72464962a644475d7d29d9c5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41117 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7b1317f6183249448df0505cf33da248" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38193 } health_report { overall_health: UNKNOWN } } }
W20260924 09:55:46.245546 1153 tablet.cc:2369] T 191b15005290429fb630741e7663c502 P 3a88f350ab134eaeb41c2531acfb2f90: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 24 09:55:46 dist-test-slave-w8d2 krb5kdc[918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243743, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260924 09:55:46.279829 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:55:46.266480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39206 (local address 127.0.97.1:36355)
0924 09:55:46.266688 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0924 09:55:46.266693 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 09:55:46.267141 (+ 448us) server_negotiation.cc:407] Connection header received
0924 09:55:46.267345 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:55:46.267350 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:55:46.267445 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:55:46.267548 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:55:46.268844 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:55:46.269558 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:55:46.270673 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:55:46.271063 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:55:46.271155 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:55:46.275010 (+ 3855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:55:46.275024 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:55:46.275041 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:55:46.275072 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:55:46.277577 (+ 2505us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:55:46.278164 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:55:46.278171 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:55:46.278174 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:55:46.278283 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:55:46.278540 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:55:46.278542 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:55:46.278544 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:55:46.278893 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:55:46.279096 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0924 09:55:46.279401 (+ 305us) server_negotiation.cc:299] Negotiation successful
0924 09:55:46.279556 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
Sep 24 09:55:46 dist-test-slave-w8d2 krb5kdc[918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243746, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 09:55:46 dist-test-slave-w8d2 krb5kdc[918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243746, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260924 09:55:46.321161 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:55:46.308700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39214 (local address 127.0.97.1:36355)
0924 09:55:46.308891 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0924 09:55:46.308898 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 09:55:46.309185 (+ 287us) server_negotiation.cc:407] Connection header received
0924 09:55:46.309369 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:55:46.309374 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:55:46.309434 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:55:46.309517 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:55:46.310740 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:55:46.311485 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:55:46.312687 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:55:46.313099 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:55:46.313203 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:55:46.316591 (+ 3388us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:55:46.316595 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:55:46.316598 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:55:46.316630 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:55:46.319273 (+ 2643us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:55:46.319809 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:55:46.319813 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:55:46.319815 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:55:46.319922 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:55:46.320175 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:55:46.320178 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:55:46.320180 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:55:46.320417 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:55:46.320586 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0924 09:55:46.320791 (+ 205us) server_negotiation.cc:299] Negotiation successful
0924 09:55:46.320937 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
Sep 24 09:55:46 dist-test-slave-w8d2 krb5kdc[918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243746, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 09:55:46 dist-test-slave-w8d2 krb5kdc[918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243746, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260924 09:55:46.359112 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:55:46.347112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39218 (local address 127.0.97.1:36355)
0924 09:55:46.347387 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0924 09:55:46.347393 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 09:55:46.347587 (+ 194us) server_negotiation.cc:407] Connection header received
0924 09:55:46.347733 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:55:46.347737 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:55:46.347798 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:55:46.347875 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:55:46.349042 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:55:46.349755 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:55:46.350869 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:55:46.351289 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:55:46.351386 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:55:46.354924 (+ 3538us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:55:46.354929 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:55:46.354932 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:55:46.354959 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:55:46.357095 (+ 2136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:55:46.357601 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:55:46.357605 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:55:46.357607 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:55:46.357711 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:55:46.358036 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:55:46.358039 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:55:46.358041 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:55:46.358305 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:55:46.358484 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0924 09:55:46.358657 (+ 173us) server_negotiation.cc:299] Negotiation successful
0924 09:55:46.358815 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
W20260924 09:55:46.360764 1067 tablet_service.cc:2908] Rejecting scan request for tablet 191b15005290429fb630741e7663c502: Uninitialized: clean time has not yet been initialized
I20260924 09:55:46.363444 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu pid 1024
I20260924 09:55:46.374850 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:55:46.364367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39234 (local address 127.0.97.1:36355)
0924 09:55:46.364514 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0924 09:55:46.364518 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 09:55:46.364809 (+ 291us) server_negotiation.cc:407] Connection header received
0924 09:55:46.364928 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:55:46.364932 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:55:46.364983 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:55:46.365054 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:55:46.366028 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:55:46.366707 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:55:46.368332 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:55:46.368719 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:55:46.368812 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:55:46.370664 (+ 1852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:55:46.370669 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:55:46.370672 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:55:46.370699 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:55:46.372951 (+ 2252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:55:46.373503 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:55:46.373506 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:55:46.373508 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:55:46.373604 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:55:46.373883 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:55:46.373885 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:55:46.373887 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:55:46.374122 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:55:46.374316 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0924 09:55:46.374477 (+ 161us) server_negotiation.cc:299] Negotiation successful
0924 09:55:46.374617 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 09:55:46.375481 1147 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 09:55:46.462292 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu with pid 1024
I20260924 09:55:46.475085 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu pid 1158
Sep 24 09:55:46 dist-test-slave-w8d2 krb5kdc[918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243746, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260924 09:55:46.488170 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:55:46.475797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36830 (local address 127.0.97.2:41117)
0924 09:55:46.475981 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0924 09:55:46.475985 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 09:55:46.476002 (+ 17us) server_negotiation.cc:407] Connection header received
0924 09:55:46.476141 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:55:46.476144 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:55:46.476207 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:55:46.476293 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:55:46.477312 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:55:46.477999 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:55:46.479184 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:55:46.479603 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:55:46.479690 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:55:46.483160 (+ 3470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:55:46.483172 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:55:46.483188 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:55:46.483222 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:55:46.485849 (+ 2627us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:55:46.486421 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:55:46.486428 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:55:46.486431 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:55:46.486571 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:55:46.486845 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:55:46.486848 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:55:46.486850 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:55:46.487152 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:55:46.487399 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0924 09:55:46.487653 (+ 254us) server_negotiation.cc:299] Negotiation successful
0924 09:55:46.487846 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260924 09:55:46.488790 1281 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 09:55:46.574991 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu with pid 1158
I20260924 09:55:46.587564 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu pid 1291
Sep 24 09:55:46 dist-test-slave-w8d2 krb5kdc[918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243746, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260924 09:55:46.600410 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:55:46.588232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51028 (local address 127.0.97.3:38193)
0924 09:55:46.588393 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0924 09:55:46.588397 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 09:55:46.588570 (+ 173us) server_negotiation.cc:407] Connection header received
0924 09:55:46.588767 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:55:46.588771 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:55:46.588846 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:55:46.588924 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:55:46.590082 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:55:46.590747 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:55:46.591926 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:55:46.592308 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:55:46.592394 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:55:46.595962 (+ 3568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:55:46.595976 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:55:46.595992 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:55:46.596021 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:55:46.598303 (+ 2282us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:55:46.598839 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:55:46.598845 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:55:46.598847 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:55:46.598954 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:55:46.599210 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:55:46.599213 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:55:46.599215 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:55:46.599548 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:55:46.599742 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0924 09:55:46.599941 (+ 199us) server_negotiation.cc:299] Negotiation successful
0924 09:55:46.600095 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 09:55:46.601002 1414 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 09:55:46.683717 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu with pid 1291
I20260924 09:55:46.696350 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu pid 931
I20260924 09:55:46.697369 993 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 09:55:46.801630 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu with pid 931
2026-09-24T09:55:46Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (6587 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790243735634835-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 09:55:46 dist-test-slave-w8d2 krb5kdc[1467](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 09:55:46 dist-test-slave-w8d2 krb5kdc[1467](info): set up 2 sockets
Sep 24 09:55:46 dist-test-slave-w8d2 krb5kdc[1467](info): commencing operation
krb5kdc: starting...
W20260924 09:55:47.795097 388 mini_kdc.cc:121] Time spent starting KDC: real 0.957s user 0.000s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790243735634835-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790243735634835-388-0/krb5kdc/test-user.keytab.
Sep 24 09:55:47 dist-test-slave-w8d2 krb5kdc[1467](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243747, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T09:55:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T09:55:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790243735634835-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790243735634835-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260924 09:55:47.956580 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
/tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790243735634835-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790243735634835-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790243735634835-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790243735634835-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42407
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790243735634835-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:42407
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790243735634835-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790243735634835-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790243735634835-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 09:55:48.260577 1480 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 09:55:48.260890 1480 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 09:55:48.261021 1480 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 09:55:48.268186 1480 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 09:55:48.268301 1480 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260924 09:55:48.268368 1480 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 09:55:48.268448 1480 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 09:55:48.268507 1480 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 09:55:48.268586 1480 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 09:55:48.278239 1480 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790243735634835-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790243735634835-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:42407
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790243735634835-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42407
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790243735634835-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790243735634835-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 09:55:48.279541 1480 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 09:55:48.280807 1480 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 09:55:48.291605 1486 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:55:48.291607 1488 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:55:48.291682 1485 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 09:55:48.292618 1480 server_base.cc:1034] running on GCE node
I20260924 09:55:48.293298 1480 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 09:55:48.294590 1480 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 09:55:48.295786 1480 hybrid_clock.cc:648] HybridClock initialized: now 1790243748295718 us; error 43 us; skew 500 ppm
I20260924 09:55:48.296074 1480 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 09:55:48 dist-test-slave-w8d2 krb5kdc[1467](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243748, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 09:55:48.298836 1480 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260924 09:55:48.300130 1480 webserver.cc:460] Webserver started at http://127.0.97.62:40585/ using document root <none> and password file <none>
I20260924 09:55:48.300746 1480 fs_manager.cc:359] Metadata directory not provided
I20260924 09:55:48.300823 1480 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 09:55:48.301074 1480 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 09:55:48.303800 1480 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790243735634835-388-0/minicluster-data/master-0/data/instance:
uuid: "9e23454a67c54530a74ec880bbe721d4"
format_stamp: "Formatted at 2026-09-24 09:55:48 on dist-test-slave-w8d2"
server_key: "305f22b98dc88bb139a91d6039661f70"
server_key_iv: "cfc9c1f0e6fce1d2b93e21c7e334af7f"
server_key_version: "encryptionkey@0"
I20260924 09:55:48.304490 1480 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790243735634835-388-0/minicluster-data/master-0/wal/instance:
uuid: "9e23454a67c54530a74ec880bbe721d4"
format_stamp: "Formatted at 2026-09-24 09:55:48 on dist-test-slave-w8d2"
server_key: "305f22b98dc88bb139a91d6039661f70"
server_key_iv: "cfc9c1f0e6fce1d2b93e21c7e334af7f"
server_key_version: "encryptionkey@0"
I20260924 09:55:48.309221 1480 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 09:55:48.312413 1495 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 09:55:48.313750 1480 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 09:55:48.313890 1480 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790243735634835-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790243735634835-388-0/minicluster-data/master-0/wal
uuid: "9e23454a67c54530a74ec880bbe721d4"
format_stamp: "Formatted at 2026-09-24 09:55:48 on dist-test-slave-w8d2"
server_key: "305f22b98dc88bb139a91d6039661f70"
server_key_iv: "cfc9c1f0e6fce1d2b93e21c7e334af7f"
server_key_version: "encryptionkey@0"
I20260924 09:55:48.314076 1480 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790243735634835-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790243735634835-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790243735634835-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 09:55:48.331357 1480 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 09:55:48.335357 1480 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 09:55:48.335608 1480 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 09:55:48.354884 1480 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:42407
I20260924 09:55:48.354919 1556 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:42407 every 8 connection(s)
I20260924 09:55:48.356192 1480 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790243735634835-388-0/minicluster-data/master-0/data/info.pb
I20260924 09:55:48.359882 1557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 09:55:48.360282 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu as pid 1480
I20260924 09:55:48.360486 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790243735634835-388-0/minicluster-data/master-0/wal/instance
I20260924 09:55:48.361287 388 external_mini_cluster.cc:1428] Setting key 1a750893a7e2a19b1383374a134c355a
I20260924 09:55:48.373262 1557 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e23454a67c54530a74ec880bbe721d4: Bootstrap starting.
Sep 24 09:55:48 dist-test-slave-w8d2 krb5kdc[1467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243747, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 09:55:48.376940 1557 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9e23454a67c54530a74ec880bbe721d4: Neither blocks nor log segments found. Creating new log.
I20260924 09:55:48.377970 1557 log.cc:824] T 00000000000000000000000000000000 P 9e23454a67c54530a74ec880bbe721d4: Log is configured to *not* fsync() on all Append() calls
I20260924 09:55:48.380923 1557 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e23454a67c54530a74ec880bbe721d4: No bootstrap required, opened a new log
I20260924 09:55:48.383078 1560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:55:48.363215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57826 (local address 127.0.97.62:42407)
0924 09:55:48.364001 (+ 786us) server_negotiation.cc:206] Beginning negotiation
0924 09:55:48.364016 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 09:55:48.364056 (+ 40us) server_negotiation.cc:407] Connection header received
0924 09:55:48.365961 (+ 1905us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:55:48.365984 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:55:48.366378 (+ 394us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:55:48.366705 (+ 327us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:55:48.368284 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:55:48.369518 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:55:48.371193 (+ 1675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:55:48.371859 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:55:48.371993 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:55:48.376464 (+ 4471us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:55:48.376472 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:55:48.376488 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:55:48.376522 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:55:48.379379 (+ 2857us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:55:48.380097 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:55:48.380102 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:55:48.380108 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:55:48.380263 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:55:48.380626 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:55:48.380629 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:55:48.380631 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:55:48.381144 (+ 513us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:55:48.381342 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0924 09:55:48.381749 (+ 407us) server_negotiation.cc:299] Negotiation successful
0924 09:55:48.382074 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":542,"thread_start_us":337,"threads_started":1}
I20260924 09:55:48.388808 1557 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9e23454a67c54530a74ec880bbe721d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e23454a67c54530a74ec880bbe721d4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42407 } }
I20260924 09:55:48.389130 1557 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9e23454a67c54530a74ec880bbe721d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 09:55:48.389214 1557 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9e23454a67c54530a74ec880bbe721d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e23454a67c54530a74ec880bbe721d4, State: Initialized, Role: FOLLOWER
I20260924 09:55:48.389788 1557 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9e23454a67c54530a74ec880bbe721d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e23454a67c54530a74ec880bbe721d4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42407 } }
I20260924 09:55:48.390049 1557 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9e23454a67c54530a74ec880bbe721d4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 09:55:48.390168 1557 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9e23454a67c54530a74ec880bbe721d4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 09:55:48.390313 1557 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9e23454a67c54530a74ec880bbe721d4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 09:55:48.392431 1557 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9e23454a67c54530a74ec880bbe721d4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e23454a67c54530a74ec880bbe721d4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42407 } }
I20260924 09:55:48.392889 1557 leader_election.cc:302] T 00000000000000000000000000000000 P 9e23454a67c54530a74ec880bbe721d4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9e23454a67c54530a74ec880bbe721d4; no voters:
I20260924 09:55:48.393417 1557 leader_election.cc:288] T 00000000000000000000000000000000 P 9e23454a67c54530a74ec880bbe721d4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 09:55:48.393563 1562 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9e23454a67c54530a74ec880bbe721d4 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 09:55:48.394423 1562 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9e23454a67c54530a74ec880bbe721d4 [term 1 LEADER]: Becoming Leader. State: Replica: 9e23454a67c54530a74ec880bbe721d4, State: Running, Role: LEADER
I20260924 09:55:48.394886 1562 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9e23454a67c54530a74ec880bbe721d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e23454a67c54530a74ec880bbe721d4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42407 } }
I20260924 09:55:48.395417 1557 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9e23454a67c54530a74ec880bbe721d4 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 09:55:48.397396 1563 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e23454a67c54530a74ec880bbe721d4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9e23454a67c54530a74ec880bbe721d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e23454a67c54530a74ec880bbe721d4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42407 } } }
I20260924 09:55:48.397688 1563 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e23454a67c54530a74ec880bbe721d4 [sys.catalog]: This master's current role is: LEADER
I20260924 09:55:48.398425 1570 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 09:55:48.399662 1564 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e23454a67c54530a74ec880bbe721d4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9e23454a67c54530a74ec880bbe721d4. Latest consensus state: current_term: 1 leader_uuid: "9e23454a67c54530a74ec880bbe721d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e23454a67c54530a74ec880bbe721d4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42407 } } }
I20260924 09:55:48.399960 1564 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e23454a67c54530a74ec880bbe721d4 [sys.catalog]: This master's current role is: LEADER
I20260924 09:55:48.403640 1570 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 09:55:48.412493 1570 catalog_manager.cc:1347] Generated new cluster ID: c4a75b40fca3412aa3a49cba2960bd90
I20260924 09:55:48.412604 1570 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 09:55:48.432351 1570 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 09:55:48.434142 1570 catalog_manager.cc:1504] Loading token signing keys...
I20260924 09:55:48.444628 1570 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9e23454a67c54530a74ec880bbe721d4: Generated new TSK 0
I20260924 09:55:48.445643 1570 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790243735634835-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790243735634835-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260924 09:55:48.482661 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
/tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790243735634835-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790243735634835-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790243735634835-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790243735634835-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42407
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790243735634835-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790243735634835-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790243735634835-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790243735634835-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 09:55:48.791127 1583 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 09:55:48.791450 1583 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 09:55:48.791592 1583 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 09:55:48.798784 1583 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 09:55:48.798894 1583 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 09:55:48.799016 1583 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 09:55:48.799117 1583 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260924 09:55:48.808796 1583 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790243735634835-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790243735634835-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790243735634835-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790243735634835-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:42407
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790243735634835-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 09:55:48.809993 1583 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 09:55:48.811275 1583 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 09:55:48.823545 1588 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:55:48.823472 1591 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:55:48.823472 1589 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 09:55:48.824450 1583 server_base.cc:1034] running on GCE node
I20260924 09:55:48.824998 1583 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 09:55:48.825810 1583 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 09:55:48.827008 1583 hybrid_clock.cc:648] HybridClock initialized: now 1790243748826943 us; error 40 us; skew 500 ppm
I20260924 09:55:48.827323 1583 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 09:55:48 dist-test-slave-w8d2 krb5kdc[1467](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243748, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 09:55:48.830192 1583 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260924 09:55:48.831565 1583 webserver.cc:460] Webserver started at http://127.0.97.1:34399/ using document root <none> and password file <none>
I20260924 09:55:48.832216 1583 fs_manager.cc:359] Metadata directory not provided
I20260924 09:55:48.832307 1583 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 09:55:48.832602 1583 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 09:55:48.835493 1583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790243735634835-388-0/minicluster-data/ts-0/data/instance:
uuid: "50155961724d43b585d6e12c12bab09a"
format_stamp: "Formatted at 2026-09-24 09:55:48 on dist-test-slave-w8d2"
server_key: "5cf28a51ab86b51a8f1daa0afebcbda6"
server_key_iv: "999173b277215142390e1f5c1138ece9"
server_key_version: "encryptionkey@0"
I20260924 09:55:48.836233 1583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790243735634835-388-0/minicluster-data/ts-0/wal/instance:
uuid: "50155961724d43b585d6e12c12bab09a"
format_stamp: "Formatted at 2026-09-24 09:55:48 on dist-test-slave-w8d2"
server_key: "5cf28a51ab86b51a8f1daa0afebcbda6"
server_key_iv: "999173b277215142390e1f5c1138ece9"
server_key_version: "encryptionkey@0"
I20260924 09:55:48.841344 1583 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 09:55:48.844671 1598 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 09:55:48.845877 1583 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 09:55:48.846027 1583 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790243735634835-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790243735634835-388-0/minicluster-data/ts-0/wal
uuid: "50155961724d43b585d6e12c12bab09a"
format_stamp: "Formatted at 2026-09-24 09:55:48 on dist-test-slave-w8d2"
server_key: "5cf28a51ab86b51a8f1daa0afebcbda6"
server_key_iv: "999173b277215142390e1f5c1138ece9"
server_key_version: "encryptionkey@0"
I20260924 09:55:48.846212 1583 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790243735634835-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790243735634835-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790243735634835-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 09:55:48.865677 1583 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 09:55:48.869417 1583 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 09:55:48.869661 1583 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 09:55:48.872246 1583 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 09:55:48.872344 1583 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 09:55:48.872479 1583 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 09:55:48.872547 1583 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 09:55:48 dist-test-slave-w8d2 krb5kdc[1467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243748, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 09:55:48.893247 1560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:55:48.872615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:47077 (local address 127.0.97.62:42407)
0924 09:55:48.872769 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0924 09:55:48.872775 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 09:55:48.874202 (+ 1427us) server_negotiation.cc:407] Connection header received
0924 09:55:48.876234 (+ 2032us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:55:48.876240 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:55:48.876321 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:55:48.876405 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:55:48.878407 (+ 2002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:55:48.879375 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:55:48.881710 (+ 2335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:55:48.882205 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:55:48.882328 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:55:48.886709 (+ 4381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:55:48.886717 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:55:48.886722 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:55:48.886767 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:55:48.889731 (+ 2964us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:55:48.890634 (+ 903us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:55:48.890639 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:55:48.890642 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:55:48.890804 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:55:48.891389 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:55:48.891394 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:55:48.891397 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:55:48.891756 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:55:48.891883 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0924 09:55:48.892718 (+ 835us) server_negotiation.cc:299] Negotiation successful
0924 09:55:48.892920 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260924 09:55:48.894306 1607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:55:48.873303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42407 (local address 127.0.97.1:47077)
0924 09:55:48.874056 (+ 753us) negotiation.cc:107] Waiting for socket to connect
0924 09:55:48.874098 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 09:55:48.875943 (+ 1845us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:55:48.876673 (+ 730us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:55:48.876681 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:55:48.877395 (+ 714us) client_negotiation.cc:189] Negotiated authn=SASL
0924 09:55:48.878238 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:55:48.878256 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:55:48.879535 (+ 1279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:55:48.879539 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:55:48.881579 (+ 2040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:55:48.881590 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:55:48.882418 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:55:48.882425 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:55:48.882739 (+ 314us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:55:48.883503 (+ 764us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 09:55:48.883558 (+ 55us) client_negotiation.cc:656] calling sasl_client_start()
0924 09:55:48.886437 (+ 2879us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 09:55:48.889861 (+ 3424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 09:55:48.889870 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 09:55:48.889890 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0924 09:55:48.890503 (+ 613us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 09:55:48.890944 (+ 441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 09:55:48.890950 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 09:55:48.890954 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 09:55:48.891283 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 09:55:48.891881 (+ 598us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 09:55:48.891888 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 09:55:48.892291 (+ 403us) client_negotiation.cc:769] Sending connection context
0924 09:55:48.892625 (+ 334us) client_negotiation.cc:240] Negotiation successful
0924 09:55:48.893157 (+ 532us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":335,"threads_started":1}
I20260924 09:55:48.912690 1560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:55:48.899811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:41153 (local address 127.0.97.62:42407)
0924 09:55:48.900017 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0924 09:55:48.900023 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 09:55:48.900475 (+ 452us) server_negotiation.cc:407] Connection header received
0924 09:55:48.900698 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:55:48.900704 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:55:48.900783 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:55:48.900886 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:55:48.901669 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:55:48.902820 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:55:48.904341 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:55:48.904972 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:55:48.905093 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:55:48.906675 (+ 1582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:55:48.906683 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:55:48.906688 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:55:48.906724 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:55:48.909917 (+ 3193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:55:48.910700 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:55:48.910705 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:55:48.910708 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:55:48.910854 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:55:48.911173 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:55:48.911178 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:55:48.911182 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:55:48.911552 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:55:48.911704 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0924 09:55:48.912141 (+ 437us) server_negotiation.cc:299] Negotiation successful
0924 09:55:48.912345 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 09:55:48.912777 1673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:55:48.899797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42407 (local address 127.0.97.1:41153)
0924 09:55:48.900377 (+ 580us) negotiation.cc:107] Waiting for socket to connect
0924 09:55:48.900392 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 09:55:48.900546 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:55:48.900898 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:55:48.900901 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:55:48.901292 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0924 09:55:48.901555 (+ 263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:55:48.901563 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:55:48.902931 (+ 1368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:55:48.902934 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:55:48.904148 (+ 1214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:55:48.904158 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:55:48.905163 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:55:48.905170 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:55:48.905301 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:55:48.905837 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 09:55:48.905852 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 09:55:48.906509 (+ 657us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 09:55:48.910066 (+ 3557us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 09:55:48.910071 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 09:55:48.910075 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 09:55:48.910585 (+ 510us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 09:55:48.910953 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 09:55:48.910956 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 09:55:48.910958 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 09:55:48.911080 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 09:55:48.911665 (+ 585us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 09:55:48.911670 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 09:55:48.911894 (+ 224us) client_negotiation.cc:769] Sending connection context
0924 09:55:48.912056 (+ 162us) client_negotiation.cc:240] Negotiation successful
0924 09:55:48.912373 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":305,"threads_started":1}
I20260924 09:55:48.921221 1583 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:44123
I20260924 09:55:48.921248 1717 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:44123 every 8 connection(s)
I20260924 09:55:48.922379 1583 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790243735634835-388-0/minicluster-data/ts-0/data/info.pb
I20260924 09:55:48.924154 1718 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42407
I20260924 09:55:48.924355 1718 heartbeater.cc:464] Registering TS with master...
I20260924 09:55:48.924952 1718 heartbeater.cc:511] Master 127.0.97.62:42407 requested a full tablet report, sending...
I20260924 09:55:48.926712 1512 ts_manager.cc:194] Registered new tserver with Master: 50155961724d43b585d6e12c12bab09a (127.0.97.1:44123)
I20260924 09:55:48.928099 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu as pid 1583
I20260924 09:55:48.928128 1512 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:47077
I20260924 09:55:48.928269 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790243735634835-388-0/minicluster-data/ts-0/wal/instance
I20260924 09:55:48.928889 388 external_mini_cluster.cc:1428] Setting key 76d8a07b81ac9f30a5378020d496978c
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790243735634835-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790243735634835-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260924 09:55:48.962545 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
/tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790243735634835-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790243735634835-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790243735634835-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790243735634835-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42407
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790243735634835-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790243735634835-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790243735634835-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790243735634835-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 09:55:49.266013 1722 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 09:55:49.266281 1722 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 09:55:49.266403 1722 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 09:55:49.273420 1722 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 09:55:49.273548 1722 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 09:55:49.273669 1722 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 09:55:49.273775 1722 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260924 09:55:49.283217 1722 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790243735634835-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790243735634835-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790243735634835-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790243735634835-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:42407
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790243735634835-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 09:55:49.284442 1722 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 09:55:49.285737 1722 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 09:55:49.298223 1727 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:55:49.298293 1728 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:55:49.298249 1730 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 09:55:49.304625 1722 server_base.cc:1034] running on GCE node
I20260924 09:55:49.305194 1722 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 09:55:49.306085 1722 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 09:55:49.307323 1722 hybrid_clock.cc:648] HybridClock initialized: now 1790243749307243 us; error 50 us; skew 500 ppm
I20260924 09:55:49.307616 1722 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 09:55:49 dist-test-slave-w8d2 krb5kdc[1467](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243749, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 09:55:49.310478 1722 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260924 09:55:49.311944 1722 webserver.cc:460] Webserver started at http://127.0.97.2:37133/ using document root <none> and password file <none>
I20260924 09:55:49.312634 1722 fs_manager.cc:359] Metadata directory not provided
I20260924 09:55:49.312728 1722 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 09:55:49.313024 1722 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 09:55:49.315886 1722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790243735634835-388-0/minicluster-data/ts-1/data/instance:
uuid: "7bda7656bed9450db86559665a6cc3fd"
format_stamp: "Formatted at 2026-09-24 09:55:49 on dist-test-slave-w8d2"
server_key: "ae97c421b4bdd5fd2e51627d0434486a"
server_key_iv: "d4661268f91fbda378cd5b9949b4e828"
server_key_version: "encryptionkey@0"
I20260924 09:55:49.316649 1722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790243735634835-388-0/minicluster-data/ts-1/wal/instance:
uuid: "7bda7656bed9450db86559665a6cc3fd"
format_stamp: "Formatted at 2026-09-24 09:55:49 on dist-test-slave-w8d2"
server_key: "ae97c421b4bdd5fd2e51627d0434486a"
server_key_iv: "d4661268f91fbda378cd5b9949b4e828"
server_key_version: "encryptionkey@0"
I20260924 09:55:49.321575 1722 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260924 09:55:49.324908 1737 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 09:55:49.326146 1722 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 09:55:49.326287 1722 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790243735634835-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790243735634835-388-0/minicluster-data/ts-1/wal
uuid: "7bda7656bed9450db86559665a6cc3fd"
format_stamp: "Formatted at 2026-09-24 09:55:49 on dist-test-slave-w8d2"
server_key: "ae97c421b4bdd5fd2e51627d0434486a"
server_key_iv: "d4661268f91fbda378cd5b9949b4e828"
server_key_version: "encryptionkey@0"
I20260924 09:55:49.326485 1722 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790243735634835-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790243735634835-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790243735634835-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 09:55:49.340790 1722 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 09:55:49.344851 1722 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 09:55:49.345101 1722 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 09:55:49.347716 1722 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 09:55:49.347829 1722 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 09:55:49.347949 1722 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 09:55:49.348016 1722 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 09:55:49 dist-test-slave-w8d2 krb5kdc[1467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243749, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 09:55:49.370502 1560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:55:49.348323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:59161 (local address 127.0.97.62:42407)
0924 09:55:49.348522 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0924 09:55:49.348528 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 09:55:49.350085 (+ 1557us) server_negotiation.cc:407] Connection header received
0924 09:55:49.352219 (+ 2134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:55:49.352225 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:55:49.352322 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:55:49.352418 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:55:49.355808 (+ 3390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:55:49.356965 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:55:49.359568 (+ 2603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:55:49.360278 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:55:49.360420 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:55:49.364375 (+ 3955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:55:49.364382 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:55:49.364387 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:55:49.364439 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:55:49.367504 (+ 3065us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:55:49.368342 (+ 838us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:55:49.368347 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:55:49.368351 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:55:49.368464 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:55:49.368847 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:55:49.368851 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:55:49.368853 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:55:49.369133 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:55:49.369277 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0924 09:55:49.369953 (+ 676us) server_negotiation.cc:299] Negotiation successful
0924 09:55:49.370174 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260924 09:55:49.371769 1746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:55:49.349021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42407 (local address 127.0.97.2:59161)
0924 09:55:49.349911 (+ 890us) negotiation.cc:107] Waiting for socket to connect
0924 09:55:49.349958 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0924 09:55:49.351928 (+ 1970us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:55:49.352804 (+ 876us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:55:49.352818 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:55:49.353707 (+ 889us) client_negotiation.cc:189] Negotiated authn=SASL
0924 09:55:49.355570 (+ 1863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:55:49.355597 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:55:49.357174 (+ 1577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:55:49.357180 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:55:49.359354 (+ 2174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:55:49.359388 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:55:49.360667 (+ 1279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:55:49.360674 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:55:49.360978 (+ 304us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:55:49.361776 (+ 798us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 09:55:49.361813 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0924 09:55:49.364125 (+ 2312us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 09:55:49.367676 (+ 3551us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 09:55:49.367685 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 09:55:49.367704 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0924 09:55:49.368217 (+ 513us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 09:55:49.368561 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 09:55:49.368564 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 09:55:49.368566 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 09:55:49.368733 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 09:55:49.369251 (+ 518us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 09:55:49.369258 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 09:55:49.369603 (+ 345us) client_negotiation.cc:769] Sending connection context
0924 09:55:49.369833 (+ 230us) client_negotiation.cc:240] Negotiation successful
0924 09:55:49.370335 (+ 502us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":593,"thread_start_us":379,"threads_started":1}
I20260924 09:55:49.399403 1560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:55:49.381544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:34431 (local address 127.0.97.62:42407)
0924 09:55:49.381714 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0924 09:55:49.381720 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 09:55:49.382488 (+ 768us) server_negotiation.cc:407] Connection header received
0924 09:55:49.382683 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:55:49.382690 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:55:49.382772 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:55:49.382879 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:55:49.384069 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:55:49.385156 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:55:49.387030 (+ 1874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:55:49.387589 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:55:49.387694 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:55:49.394090 (+ 6396us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:55:49.394095 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:55:49.394098 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:55:49.394127 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:55:49.397015 (+ 2888us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:55:49.397651 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:55:49.397657 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:55:49.397660 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:55:49.397769 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:55:49.398068 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:55:49.398073 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:55:49.398076 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:55:49.398357 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:55:49.398477 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0924 09:55:49.398840 (+ 363us) server_negotiation.cc:299] Negotiation successful
0924 09:55:49.399037 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 09:55:49.399649 1805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:55:49.381631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42407 (local address 127.0.97.2:34431)
0924 09:55:49.382388 (+ 757us) negotiation.cc:107] Waiting for socket to connect
0924 09:55:49.382406 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 09:55:49.382547 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:55:49.382935 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:55:49.382938 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:55:49.383501 (+ 563us) client_negotiation.cc:189] Negotiated authn=SASL
0924 09:55:49.383885 (+ 384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:55:49.383902 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:55:49.385294 (+ 1392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:55:49.385299 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:55:49.386905 (+ 1606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:55:49.386920 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:55:49.387771 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:55:49.387778 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:55:49.387933 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:55:49.392800 (+ 4867us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 09:55:49.392818 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 09:55:49.393866 (+ 1048us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 09:55:49.397151 (+ 3285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 09:55:49.397157 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 09:55:49.397161 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 09:55:49.397549 (+ 388us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 09:55:49.397861 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 09:55:49.397866 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 09:55:49.397870 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 09:55:49.397980 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 09:55:49.398460 (+ 480us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 09:55:49.398465 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 09:55:49.398629 (+ 164us) client_negotiation.cc:769] Sending connection context
0924 09:55:49.398739 (+ 110us) client_negotiation.cc:240] Negotiation successful
0924 09:55:49.399160 (+ 421us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":599,"thread_start_us":444,"threads_started":1}
I20260924 09:55:49.403512 1722 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:34007
I20260924 09:55:49.403532 1856 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:34007 every 8 connection(s)
I20260924 09:55:49.404652 1722 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790243735634835-388-0/minicluster-data/ts-1/data/info.pb
I20260924 09:55:49.406222 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu as pid 1722
I20260924 09:55:49.406368 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790243735634835-388-0/minicluster-data/ts-1/wal/instance
I20260924 09:55:49.406422 1857 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42407
I20260924 09:55:49.406617 1857 heartbeater.cc:464] Registering TS with master...
I20260924 09:55:49.406823 388 external_mini_cluster.cc:1428] Setting key 84bdee0b9e97ffd7047b48572e1e6240
I20260924 09:55:49.407218 1857 heartbeater.cc:511] Master 127.0.97.62:42407 requested a full tablet report, sending...
I20260924 09:55:49.408456 1512 ts_manager.cc:194] Registered new tserver with Master: 7bda7656bed9450db86559665a6cc3fd (127.0.97.2:34007)
I20260924 09:55:49.409228 1512 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:59161
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790243735634835-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790243735634835-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260924 09:55:49.441927 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
/tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790243735634835-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790243735634835-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790243735634835-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790243735634835-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42407
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790243735634835-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790243735634835-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790243735634835-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790243735634835-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 09:55:49.744796 1861 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 09:55:49.745078 1861 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 09:55:49.745200 1861 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 09:55:49.752341 1861 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 09:55:49.752485 1861 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 09:55:49.752610 1861 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 09:55:49.752709 1861 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260924 09:55:49.762303 1861 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790243735634835-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790243735634835-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790243735634835-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790243735634835-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:42407
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790243735634835-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 09:55:49.763566 1861 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 09:55:49.764873 1861 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 09:55:49.776934 1866 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:55:49.776942 1867 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:55:49.777489 1869 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 09:55:49.778316 1861 server_base.cc:1034] running on GCE node
I20260924 09:55:49.778883 1861 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 09:55:49.779790 1861 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 09:55:49.780992 1861 hybrid_clock.cc:648] HybridClock initialized: now 1790243749780923 us; error 43 us; skew 500 ppm
I20260924 09:55:49.781265 1861 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 09:55:49 dist-test-slave-w8d2 krb5kdc[1467](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243749, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 09:55:49.784106 1861 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260924 09:55:49.785449 1861 webserver.cc:460] Webserver started at http://127.0.97.3:38527/ using document root <none> and password file <none>
I20260924 09:55:49.786088 1861 fs_manager.cc:359] Metadata directory not provided
I20260924 09:55:49.786190 1861 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 09:55:49.786474 1861 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 09:55:49.789265 1861 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790243735634835-388-0/minicluster-data/ts-2/data/instance:
uuid: "e2fbec6f2e574d4bbfcf0fdf050449c5"
format_stamp: "Formatted at 2026-09-24 09:55:49 on dist-test-slave-w8d2"
server_key: "89dcf46c690a042f81fb8c2112850346"
server_key_iv: "e2ada74bcec6ac3dc134145a40098257"
server_key_version: "encryptionkey@0"
I20260924 09:55:49.789992 1861 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790243735634835-388-0/minicluster-data/ts-2/wal/instance:
uuid: "e2fbec6f2e574d4bbfcf0fdf050449c5"
format_stamp: "Formatted at 2026-09-24 09:55:49 on dist-test-slave-w8d2"
server_key: "89dcf46c690a042f81fb8c2112850346"
server_key_iv: "e2ada74bcec6ac3dc134145a40098257"
server_key_version: "encryptionkey@0"
I20260924 09:55:49.795050 1861 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260924 09:55:49.798409 1876 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 09:55:49.799711 1861 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 09:55:49.799849 1861 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790243735634835-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790243735634835-388-0/minicluster-data/ts-2/wal
uuid: "e2fbec6f2e574d4bbfcf0fdf050449c5"
format_stamp: "Formatted at 2026-09-24 09:55:49 on dist-test-slave-w8d2"
server_key: "89dcf46c690a042f81fb8c2112850346"
server_key_iv: "e2ada74bcec6ac3dc134145a40098257"
server_key_version: "encryptionkey@0"
I20260924 09:55:49.800029 1861 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790243735634835-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790243735634835-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790243735634835-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 09:55:49.820874 1861 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 09:55:49.824891 1861 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 09:55:49.825150 1861 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 09:55:49.827693 1861 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 09:55:49.827790 1861 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 09:55:49.827917 1861 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 09:55:49.827983 1861 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 09:55:49 dist-test-slave-w8d2 krb5kdc[1467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243749, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 09:55:49.849661 1560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:55:49.828198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:54485 (local address 127.0.97.62:42407)
0924 09:55:49.828346 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0924 09:55:49.828350 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 09:55:49.829944 (+ 1594us) server_negotiation.cc:407] Connection header received
0924 09:55:49.832047 (+ 2103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:55:49.832053 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:55:49.832142 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:55:49.832241 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:55:49.834860 (+ 2619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:55:49.835740 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:55:49.838070 (+ 2330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:55:49.838624 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:55:49.838744 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:55:49.842724 (+ 3980us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:55:49.842732 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:55:49.842738 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:55:49.842785 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:55:49.846061 (+ 3276us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:55:49.846951 (+ 890us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:55:49.846955 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:55:49.846957 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:55:49.847079 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:55:49.847540 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:55:49.847545 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:55:49.847549 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:55:49.847866 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:55:49.848046 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0924 09:55:49.849023 (+ 977us) server_negotiation.cc:299] Negotiation successful
0924 09:55:49.849274 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260924 09:55:49.850582 1885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:55:49.828926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42407 (local address 127.0.97.3:54485)
0924 09:55:49.829732 (+ 806us) negotiation.cc:107] Waiting for socket to connect
0924 09:55:49.829837 (+ 105us) client_negotiation.cc:174] Beginning negotiation
0924 09:55:49.831781 (+ 1944us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:55:49.832540 (+ 759us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:55:49.832553 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:55:49.833588 (+ 1035us) client_negotiation.cc:189] Negotiated authn=SASL
0924 09:55:49.834651 (+ 1063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:55:49.834684 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:55:49.835878 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:55:49.835887 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:55:49.837928 (+ 2041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:55:49.837942 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:55:49.839050 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:55:49.839055 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:55:49.839316 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:55:49.840054 (+ 738us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 09:55:49.840098 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0924 09:55:49.842480 (+ 2382us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 09:55:49.846202 (+ 3722us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 09:55:49.846212 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 09:55:49.846232 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0924 09:55:49.846843 (+ 611us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 09:55:49.847167 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 09:55:49.847172 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 09:55:49.847176 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 09:55:49.847428 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 09:55:49.848047 (+ 619us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 09:55:49.848056 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 09:55:49.848554 (+ 498us) client_negotiation.cc:769] Sending connection context
0924 09:55:49.848897 (+ 343us) client_negotiation.cc:240] Negotiation successful
0924 09:55:49.849385 (+ 488us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":547,"thread_start_us":344,"threads_started":1}
I20260924 09:55:49.869853 1940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:55:49.855856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42407 (local address 127.0.97.3:46627)
0924 09:55:49.856493 (+ 637us) negotiation.cc:107] Waiting for socket to connect
0924 09:55:49.856514 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 09:55:49.856660 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:55:49.856930 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:55:49.856935 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:55:49.857495 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0924 09:55:49.857868 (+ 373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:55:49.857881 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:55:49.858796 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:55:49.858801 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:55:49.860470 (+ 1669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:55:49.860486 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:55:49.861173 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:55:49.861179 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:55:49.861391 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:55:49.862061 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 09:55:49.862090 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0924 09:55:49.863266 (+ 1176us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 09:55:49.866386 (+ 3120us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 09:55:49.866390 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 09:55:49.866394 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 09:55:49.867078 (+ 684us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 09:55:49.867552 (+ 474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 09:55:49.867557 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 09:55:49.867562 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 09:55:49.867731 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 09:55:49.868465 (+ 734us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 09:55:49.868473 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 09:55:49.868701 (+ 228us) client_negotiation.cc:769] Sending connection context
0924 09:55:49.868864 (+ 163us) client_negotiation.cc:240] Negotiation successful
0924 09:55:49.869270 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":342,"threads_started":1}
I20260924 09:55:49.870939 1560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:55:49.855552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:46627 (local address 127.0.97.62:42407)
0924 09:55:49.855733 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0924 09:55:49.855738 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 09:55:49.856597 (+ 859us) server_negotiation.cc:407] Connection header received
0924 09:55:49.856757 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:55:49.856761 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:55:49.856819 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:55:49.856890 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:55:49.857996 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:55:49.858679 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:55:49.860602 (+ 1923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:55:49.861010 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:55:49.861103 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:55:49.863436 (+ 2333us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:55:49.863441 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:55:49.863444 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:55:49.863472 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:55:49.866262 (+ 2790us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:55:49.867199 (+ 937us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:55:49.867204 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:55:49.867208 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:55:49.867406 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:55:49.867873 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:55:49.867878 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:55:49.867881 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:55:49.868251 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:55:49.868398 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0924 09:55:49.870297 (+ 1899us) server_negotiation.cc:299] Negotiation successful
0924 09:55:49.870588 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 09:55:49.880213 1861 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:35531
I20260924 09:55:49.880237 1995 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:35531 every 8 connection(s)
I20260924 09:55:49.881356 1861 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790243735634835-388-0/minicluster-data/ts-2/data/info.pb
I20260924 09:55:49.883281 1996 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42407
I20260924 09:55:49.883498 1996 heartbeater.cc:464] Registering TS with master...
I20260924 09:55:49.884100 1996 heartbeater.cc:511] Master 127.0.97.62:42407 requested a full tablet report, sending...
I20260924 09:55:49.885377 1512 ts_manager.cc:194] Registered new tserver with Master: e2fbec6f2e574d4bbfcf0fdf050449c5 (127.0.97.3:35531)
I20260924 09:55:49.886170 1512 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:54485
I20260924 09:55:49.886801 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu as pid 1861
I20260924 09:55:49.886940 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790243735634835-388-0/minicluster-data/ts-2/wal/instance
I20260924 09:55:49.887421 388 external_mini_cluster.cc:1428] Setting key a3f6de4643202e05abd1a60b38af296c
I20260924 09:55:49.890066 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 09:55:49 dist-test-slave-w8d2 krb5kdc[1467](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243749, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 09:55:49 dist-test-slave-w8d2 krb5kdc[1467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243749, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 09:55:49.930773 1718 heartbeater.cc:503] Master 127.0.97.62:42407 was elected leader, sending a full tablet report...
I20260924 09:55:49.931311 1560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:55:49.917254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57838 (local address 127.0.97.62:42407)
0924 09:55:49.917475 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0924 09:55:49.917481 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 09:55:49.917689 (+ 208us) server_negotiation.cc:407] Connection header received
0924 09:55:49.917856 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:55:49.917860 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:55:49.917952 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:55:49.918036 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:55:49.919334 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:55:49.920023 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:55:49.921491 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:55:49.921915 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:55:49.922014 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:55:49.926121 (+ 4107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:55:49.926126 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:55:49.926129 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:55:49.926163 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:55:49.928503 (+ 2340us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:55:49.929089 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:55:49.929095 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:55:49.929098 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:55:49.929210 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:55:49.929552 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:55:49.929557 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:55:49.929560 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:55:49.929814 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:55:49.930067 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0924 09:55:49.930300 (+ 233us) server_negotiation.cc:299] Negotiation successful
0924 09:55:49.930466 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260924 09:55:49.935158 1511 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57838:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 09:55:49.941973 1511 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 09:55:49.976156 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:55:49.954109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:34007 (local address 127.0.0.1:59360)
0924 09:55:49.967500 (+ 13391us) negotiation.cc:107] Waiting for socket to connect
0924 09:55:49.967521 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 09:55:49.967665 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:55:49.968562 (+ 897us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:55:49.968570 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:55:49.968618 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 09:55:49.969219 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:55:49.969234 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:55:49.970555 (+ 1321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:55:49.970560 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:55:49.973356 (+ 2796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:55:49.973371 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:55:49.975143 (+ 1772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:55:49.975149 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:55:49.975357 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:55:49.975406 (+ 49us) client_negotiation.cc:769] Sending connection context
0924 09:55:49.975523 (+ 117us) client_negotiation.cc:240] Negotiation successful
0924 09:55:49.975708 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13267,"mutex_wait_us":22,"spinlock_wait_cycles":5888,"thread_start_us":12612,"threads_started":1}
I20260924 09:55:49.976156 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:55:49.954330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59360 (local address 127.0.97.2:34007)
0924 09:55:49.954836 (+ 506us) server_negotiation.cc:206] Beginning negotiation
0924 09:55:49.954845 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 09:55:49.967633 (+ 12788us) server_negotiation.cc:407] Connection header received
0924 09:55:49.967825 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:55:49.967834 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:55:49.968104 (+ 270us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:55:49.968313 (+ 209us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 09:55:49.969415 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:55:49.970424 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:55:49.973542 (+ 3118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:55:49.974959 (+ 1417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:55:49.975076 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:55:49.975628 (+ 552us) server_negotiation.cc:1035] Waiting for connection context
0924 09:55:49.975711 (+ 83us) server_negotiation.cc:299] Negotiation successful
0924 09:55:49.975865 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":644,"threads_started":1}
I20260924 09:55:49.977196 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:55:49.954704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:35531 (local address 127.0.0.1:39216)
0924 09:55:49.968002 (+ 13298us) negotiation.cc:107] Waiting for socket to connect
0924 09:55:49.968016 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 09:55:49.968144 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:55:49.968963 (+ 819us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:55:49.968968 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:55:49.968988 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 09:55:49.969526 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:55:49.969539 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:55:49.972108 (+ 2569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:55:49.972117 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:55:49.974498 (+ 2381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:55:49.974511 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:55:49.976457 (+ 1946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:55:49.976463 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:55:49.976611 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:55:49.976656 (+ 45us) client_negotiation.cc:769] Sending connection context
0924 09:55:49.976732 (+ 76us) client_negotiation.cc:240] Negotiation successful
0924 09:55:49.976844 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13202,"mutex_wait_us":354,"thread_start_us":11133,"threads_started":1}
I20260924 09:55:49.977226 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:55:49.955184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39216 (local address 127.0.97.3:35531)
0924 09:55:49.967801 (+ 12617us) server_negotiation.cc:206] Beginning negotiation
0924 09:55:49.967811 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 09:55:49.968117 (+ 306us) server_negotiation.cc:407] Connection header received
0924 09:55:49.968289 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:55:49.968297 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:55:49.968525 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:55:49.968701 (+ 176us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 09:55:49.969710 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:55:49.971254 (+ 1544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:55:49.974680 (+ 3426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:55:49.976000 (+ 1320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:55:49.976110 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:55:49.976308 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0924 09:55:49.976787 (+ 479us) server_negotiation.cc:299] Negotiation successful
0924 09:55:49.976935 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12428,"thread_start_us":12240,"threads_started":1}
I20260924 09:55:49.977382 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:55:49.953930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44123 (local address 127.0.0.1:57902)
0924 09:55:49.965277 (+ 11347us) negotiation.cc:107] Waiting for socket to connect
0924 09:55:49.965348 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0924 09:55:49.965593 (+ 245us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:55:49.968562 (+ 2969us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:55:49.968571 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:55:49.968609 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 09:55:49.969457 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:55:49.969472 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:55:49.971301 (+ 1829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:55:49.971306 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:55:49.974284 (+ 2978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:55:49.974295 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:55:49.976042 (+ 1747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:55:49.976048 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:55:49.976218 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:55:49.976246 (+ 28us) client_negotiation.cc:769] Sending connection context
0924 09:55:49.976330 (+ 84us) client_negotiation.cc:240] Negotiation successful
0924 09:55:49.976457 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10997,"mutex_wait_us":115,"spinlock_wait_cycles":1920,"thread_start_us":14281,"threads_started":1}
I20260924 09:55:49.977968 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:55:49.954504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57902 (local address 127.0.97.1:44123)
0924 09:55:49.967857 (+ 13353us) server_negotiation.cc:206] Beginning negotiation
0924 09:55:49.967864 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 09:55:49.967888 (+ 24us) server_negotiation.cc:407] Connection header received
0924 09:55:49.967971 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:55:49.967977 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:55:49.968131 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:55:49.968313 (+ 182us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 09:55:49.969603 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:55:49.971145 (+ 1542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:55:49.974443 (+ 3298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:55:49.975618 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:55:49.975730 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:55:49.975909 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0924 09:55:49.977572 (+ 1663us) server_negotiation.cc:299] Negotiation successful
0924 09:55:49.977724 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12635,"thread_start_us":12475,"threads_started":1}
I20260924 09:55:49.980315 1787 tablet_service.cc:1467] Processing CreateTablet for tablet 209567a1c4a24d029c52b6bb3e940945 (DEFAULT_TABLE table=test-table [id=88b1bae35de34ce5a913630103050ecd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 09:55:49.981448 1787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 209567a1c4a24d029c52b6bb3e940945. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 09:55:49.981168 1648 tablet_service.cc:1467] Processing CreateTablet for tablet 209567a1c4a24d029c52b6bb3e940945 (DEFAULT_TABLE table=test-table [id=88b1bae35de34ce5a913630103050ecd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 09:55:49.982468 1648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 209567a1c4a24d029c52b6bb3e940945. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 09:55:49.982209 1926 tablet_service.cc:1467] Processing CreateTablet for tablet 209567a1c4a24d029c52b6bb3e940945 (DEFAULT_TABLE table=test-table [id=88b1bae35de34ce5a913630103050ecd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 09:55:49.983495 1926 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 209567a1c4a24d029c52b6bb3e940945. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 09:55:49.991881 2015 tablet_bootstrap.cc:492] T 209567a1c4a24d029c52b6bb3e940945 P 7bda7656bed9450db86559665a6cc3fd: Bootstrap starting.
I20260924 09:55:49.994787 2015 tablet_bootstrap.cc:654] T 209567a1c4a24d029c52b6bb3e940945 P 7bda7656bed9450db86559665a6cc3fd: Neither blocks nor log segments found. Creating new log.
I20260924 09:55:49.996026 2015 log.cc:824] T 209567a1c4a24d029c52b6bb3e940945 P 7bda7656bed9450db86559665a6cc3fd: Log is configured to *not* fsync() on all Append() calls
I20260924 09:55:49.996331 2016 tablet_bootstrap.cc:492] T 209567a1c4a24d029c52b6bb3e940945 P 50155961724d43b585d6e12c12bab09a: Bootstrap starting.
I20260924 09:55:49.999526 2015 tablet_bootstrap.cc:492] T 209567a1c4a24d029c52b6bb3e940945 P 7bda7656bed9450db86559665a6cc3fd: No bootstrap required, opened a new log
I20260924 09:55:49.999773 2016 tablet_bootstrap.cc:654] T 209567a1c4a24d029c52b6bb3e940945 P 50155961724d43b585d6e12c12bab09a: Neither blocks nor log segments found. Creating new log.
I20260924 09:55:49.999789 2015 ts_tablet_manager.cc:1397] T 209567a1c4a24d029c52b6bb3e940945 P 7bda7656bed9450db86559665a6cc3fd: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.005s
I20260924 09:55:50.000937 2016 log.cc:824] T 209567a1c4a24d029c52b6bb3e940945 P 50155961724d43b585d6e12c12bab09a: Log is configured to *not* fsync() on all Append() calls
I20260924 09:55:50.005106 2016 tablet_bootstrap.cc:492] T 209567a1c4a24d029c52b6bb3e940945 P 50155961724d43b585d6e12c12bab09a: No bootstrap required, opened a new log
I20260924 09:55:50.005615 2016 ts_tablet_manager.cc:1397] T 209567a1c4a24d029c52b6bb3e940945 P 50155961724d43b585d6e12c12bab09a: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.001s
I20260924 09:55:50.007817 2018 tablet_bootstrap.cc:492] T 209567a1c4a24d029c52b6bb3e940945 P e2fbec6f2e574d4bbfcf0fdf050449c5: Bootstrap starting.
I20260924 09:55:50.009965 2015 raft_consensus.cc:348] T 209567a1c4a24d029c52b6bb3e940945 P 7bda7656bed9450db86559665a6cc3fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50155961724d43b585d6e12c12bab09a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44123 } } peers { permanent_uuid: "7bda7656bed9450db86559665a6cc3fd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34007 } } peers { permanent_uuid: "e2fbec6f2e574d4bbfcf0fdf050449c5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35531 } }
I20260924 09:55:50.010527 2015 raft_consensus.cc:374] T 209567a1c4a24d029c52b6bb3e940945 P 7bda7656bed9450db86559665a6cc3fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 09:55:50.010687 2015 raft_consensus.cc:729] T 209567a1c4a24d029c52b6bb3e940945 P 7bda7656bed9450db86559665a6cc3fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7bda7656bed9450db86559665a6cc3fd, State: Initialized, Role: FOLLOWER
I20260924 09:55:50.011117 2018 tablet_bootstrap.cc:654] T 209567a1c4a24d029c52b6bb3e940945 P e2fbec6f2e574d4bbfcf0fdf050449c5: Neither blocks nor log segments found. Creating new log.
I20260924 09:55:50.011461 2015 consensus_queue.cc:260] T 209567a1c4a24d029c52b6bb3e940945 P 7bda7656bed9450db86559665a6cc3fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50155961724d43b585d6e12c12bab09a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44123 } } peers { permanent_uuid: "7bda7656bed9450db86559665a6cc3fd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34007 } } peers { permanent_uuid: "e2fbec6f2e574d4bbfcf0fdf050449c5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35531 } }
I20260924 09:55:50.012322 2018 log.cc:824] T 209567a1c4a24d029c52b6bb3e940945 P e2fbec6f2e574d4bbfcf0fdf050449c5: Log is configured to *not* fsync() on all Append() calls
I20260924 09:55:50.012424 1857 heartbeater.cc:503] Master 127.0.97.62:42407 was elected leader, sending a full tablet report...
I20260924 09:55:50.013047 2015 ts_tablet_manager.cc:1428] T 209567a1c4a24d029c52b6bb3e940945 P 7bda7656bed9450db86559665a6cc3fd: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
I20260924 09:55:50.015617 2018 tablet_bootstrap.cc:492] T 209567a1c4a24d029c52b6bb3e940945 P e2fbec6f2e574d4bbfcf0fdf050449c5: No bootstrap required, opened a new log
I20260924 09:55:50.015861 2018 ts_tablet_manager.cc:1397] T 209567a1c4a24d029c52b6bb3e940945 P e2fbec6f2e574d4bbfcf0fdf050449c5: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260924 09:55:50.015729 2016 raft_consensus.cc:348] T 209567a1c4a24d029c52b6bb3e940945 P 50155961724d43b585d6e12c12bab09a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50155961724d43b585d6e12c12bab09a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44123 } } peers { permanent_uuid: "7bda7656bed9450db86559665a6cc3fd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34007 } } peers { permanent_uuid: "e2fbec6f2e574d4bbfcf0fdf050449c5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35531 } }
I20260924 09:55:50.016160 2016 raft_consensus.cc:374] T 209567a1c4a24d029c52b6bb3e940945 P 50155961724d43b585d6e12c12bab09a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 09:55:50.016264 2016 raft_consensus.cc:729] T 209567a1c4a24d029c52b6bb3e940945 P 50155961724d43b585d6e12c12bab09a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50155961724d43b585d6e12c12bab09a, State: Initialized, Role: FOLLOWER
I20260924 09:55:50.016814 2016 consensus_queue.cc:260] T 209567a1c4a24d029c52b6bb3e940945 P 50155961724d43b585d6e12c12bab09a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50155961724d43b585d6e12c12bab09a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44123 } } peers { permanent_uuid: "7bda7656bed9450db86559665a6cc3fd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34007 } } peers { permanent_uuid: "e2fbec6f2e574d4bbfcf0fdf050449c5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35531 } }
I20260924 09:55:50.018467 2016 ts_tablet_manager.cc:1428] T 209567a1c4a24d029c52b6bb3e940945 P 50155961724d43b585d6e12c12bab09a: Time spent starting tablet: real 0.013s user 0.002s sys 0.011s
I20260924 09:55:50.023944 2018 raft_consensus.cc:348] T 209567a1c4a24d029c52b6bb3e940945 P e2fbec6f2e574d4bbfcf0fdf050449c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50155961724d43b585d6e12c12bab09a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44123 } } peers { permanent_uuid: "7bda7656bed9450db86559665a6cc3fd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34007 } } peers { permanent_uuid: "e2fbec6f2e574d4bbfcf0fdf050449c5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35531 } }
I20260924 09:55:50.024732 2018 raft_consensus.cc:374] T 209567a1c4a24d029c52b6bb3e940945 P e2fbec6f2e574d4bbfcf0fdf050449c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 09:55:50.024820 2018 raft_consensus.cc:729] T 209567a1c4a24d029c52b6bb3e940945 P e2fbec6f2e574d4bbfcf0fdf050449c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2fbec6f2e574d4bbfcf0fdf050449c5, State: Initialized, Role: FOLLOWER
I20260924 09:55:50.025324 2018 consensus_queue.cc:260] T 209567a1c4a24d029c52b6bb3e940945 P e2fbec6f2e574d4bbfcf0fdf050449c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50155961724d43b585d6e12c12bab09a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44123 } } peers { permanent_uuid: "7bda7656bed9450db86559665a6cc3fd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34007 } } peers { permanent_uuid: "e2fbec6f2e574d4bbfcf0fdf050449c5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35531 } }
I20260924 09:55:50.026206 1996 heartbeater.cc:503] Master 127.0.97.62:42407 was elected leader, sending a full tablet report...
I20260924 09:55:50.026687 2018 ts_tablet_manager.cc:1428] T 209567a1c4a24d029c52b6bb3e940945 P e2fbec6f2e574d4bbfcf0fdf050449c5: Time spent starting tablet: real 0.011s user 0.011s sys 0.000s
I20260924 09:55:50.053632 2020 raft_consensus.cc:482] T 209567a1c4a24d029c52b6bb3e940945 P 7bda7656bed9450db86559665a6cc3fd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 09:55:50.053915 2020 raft_consensus.cc:504] T 209567a1c4a24d029c52b6bb3e940945 P 7bda7656bed9450db86559665a6cc3fd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50155961724d43b585d6e12c12bab09a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44123 } } peers { permanent_uuid: "7bda7656bed9450db86559665a6cc3fd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34007 } } peers { permanent_uuid: "e2fbec6f2e574d4bbfcf0fdf050449c5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35531 } }
I20260924 09:55:50.055575 2020 leader_election.cc:288] T 209567a1c4a24d029c52b6bb3e940945 P 7bda7656bed9450db86559665a6cc3fd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 50155961724d43b585d6e12c12bab09a (127.0.97.1:44123), e2fbec6f2e574d4bbfcf0fdf050449c5 (127.0.97.3:35531)
I20260924 09:55:50.063036 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:55:50.055845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:36531 (local address 127.0.97.1:44123)
0924 09:55:50.056022 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0924 09:55:50.056028 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 09:55:50.056959 (+ 931us) server_negotiation.cc:407] Connection header received
0924 09:55:50.057088 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:55:50.057092 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:55:50.057155 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:55:50.057220 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 09:55:50.057712 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:55:50.058665 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:55:50.061053 (+ 2388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:55:50.062290 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:55:50.062423 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:55:50.062523 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0924 09:55:50.062715 (+ 192us) server_negotiation.cc:299] Negotiation successful
0924 09:55:50.062841 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 09:55:50.063201 2025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:55:50.055906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44123 (local address 127.0.97.2:36531)
0924 09:55:50.056871 (+ 965us) negotiation.cc:107] Waiting for socket to connect
0924 09:55:50.056883 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 09:55:50.056981 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:55:50.057259 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:55:50.057263 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:55:50.057286 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 09:55:50.057585 (+ 299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:55:50.057592 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:55:50.058768 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:55:50.058773 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:55:50.060917 (+ 2144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:55:50.060932 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:55:50.062472 (+ 1540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:55:50.062477 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:55:50.062583 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:55:50.062606 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 09:55:50.062677 (+ 71us) client_negotiation.cc:240] Negotiation successful
0924 09:55:50.062775 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":879,"mutex_wait_us":86,"thread_start_us":675,"threads_started":1}
I20260924 09:55:50.063596 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:55:50.056115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:39809 (local address 127.0.97.3:35531)
0924 09:55:50.056301 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0924 09:55:50.056307 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 09:55:50.056580 (+ 273us) server_negotiation.cc:407] Connection header received
0924 09:55:50.056794 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:55:50.056798 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:55:50.056855 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:55:50.056931 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 09:55:50.057438 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:55:50.058448 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:55:50.061058 (+ 2610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:55:50.062680 (+ 1622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:55:50.062797 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:55:50.062891 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0924 09:55:50.063270 (+ 379us) server_negotiation.cc:299] Negotiation successful
0924 09:55:50.063399 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 09:55:50.064198 2024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:55:50.055824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:35531 (local address 127.0.97.2:39809)
0924 09:55:50.056466 (+ 642us) negotiation.cc:107] Waiting for socket to connect
0924 09:55:50.056485 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 09:55:50.056645 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:55:50.056966 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:55:50.056971 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:55:50.056991 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 09:55:50.057309 (+ 318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:55:50.057320 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:55:50.058564 (+ 1244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:55:50.058567 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:55:50.060920 (+ 2353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:55:50.060931 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:55:50.062872 (+ 1941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:55:50.062878 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:55:50.063056 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:55:50.063085 (+ 29us) client_negotiation.cc:769] Sending connection context
0924 09:55:50.063763 (+ 678us) client_negotiation.cc:240] Negotiation successful
0924 09:55:50.063886 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":528,"mutex_wait_us":103,"thread_start_us":476,"threads_started":1}
I20260924 09:55:50.064251 1672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "209567a1c4a24d029c52b6bb3e940945" candidate_uuid: "7bda7656bed9450db86559665a6cc3fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50155961724d43b585d6e12c12bab09a" is_pre_election: true
I20260924 09:55:50.064791 1672 raft_consensus.cc:2454] T 209567a1c4a24d029c52b6bb3e940945 P 50155961724d43b585d6e12c12bab09a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7bda7656bed9450db86559665a6cc3fd in term 0.
I20260924 09:55:50.065133 1951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "209567a1c4a24d029c52b6bb3e940945" candidate_uuid: "7bda7656bed9450db86559665a6cc3fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2fbec6f2e574d4bbfcf0fdf050449c5" is_pre_election: true
I20260924 09:55:50.065367 1741 leader_election.cc:302] T 209567a1c4a24d029c52b6bb3e940945 P 7bda7656bed9450db86559665a6cc3fd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 50155961724d43b585d6e12c12bab09a, 7bda7656bed9450db86559665a6cc3fd; no voters:
I20260924 09:55:50.065637 1951 raft_consensus.cc:2454] T 209567a1c4a24d029c52b6bb3e940945 P e2fbec6f2e574d4bbfcf0fdf050449c5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7bda7656bed9450db86559665a6cc3fd in term 0.
I20260924 09:55:50.065763 2020 raft_consensus.cc:2781] T 209567a1c4a24d029c52b6bb3e940945 P 7bda7656bed9450db86559665a6cc3fd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 09:55:50.065896 2020 raft_consensus.cc:482] T 209567a1c4a24d029c52b6bb3e940945 P 7bda7656bed9450db86559665a6cc3fd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 09:55:50.065981 2020 raft_consensus.cc:3037] T 209567a1c4a24d029c52b6bb3e940945 P 7bda7656bed9450db86559665a6cc3fd [term 0 FOLLOWER]: Advancing to term 1
I20260924 09:55:50.068136 2020 raft_consensus.cc:504] T 209567a1c4a24d029c52b6bb3e940945 P 7bda7656bed9450db86559665a6cc3fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50155961724d43b585d6e12c12bab09a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44123 } } peers { permanent_uuid: "7bda7656bed9450db86559665a6cc3fd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34007 } } peers { permanent_uuid: "e2fbec6f2e574d4bbfcf0fdf050449c5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35531 } }
I20260924 09:55:50.068621 2020 leader_election.cc:288] T 209567a1c4a24d029c52b6bb3e940945 P 7bda7656bed9450db86559665a6cc3fd [CANDIDATE]: Term 1 election: Requested vote from peers 50155961724d43b585d6e12c12bab09a (127.0.97.1:44123), e2fbec6f2e574d4bbfcf0fdf050449c5 (127.0.97.3:35531)
I20260924 09:55:50.069017 1951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "209567a1c4a24d029c52b6bb3e940945" candidate_uuid: "7bda7656bed9450db86559665a6cc3fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2fbec6f2e574d4bbfcf0fdf050449c5"
I20260924 09:55:50.069200 1951 raft_consensus.cc:3037] T 209567a1c4a24d029c52b6bb3e940945 P e2fbec6f2e574d4bbfcf0fdf050449c5 [term 0 FOLLOWER]: Advancing to term 1
I20260924 09:55:50.069252 1672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "209567a1c4a24d029c52b6bb3e940945" candidate_uuid: "7bda7656bed9450db86559665a6cc3fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50155961724d43b585d6e12c12bab09a"
I20260924 09:55:50.069447 1672 raft_consensus.cc:3037] T 209567a1c4a24d029c52b6bb3e940945 P 50155961724d43b585d6e12c12bab09a [term 0 FOLLOWER]: Advancing to term 1
I20260924 09:55:50.071434 1951 raft_consensus.cc:2454] T 209567a1c4a24d029c52b6bb3e940945 P e2fbec6f2e574d4bbfcf0fdf050449c5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7bda7656bed9450db86559665a6cc3fd in term 1.
I20260924 09:55:50.071594 1672 raft_consensus.cc:2454] T 209567a1c4a24d029c52b6bb3e940945 P 50155961724d43b585d6e12c12bab09a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7bda7656bed9450db86559665a6cc3fd in term 1.
I20260924 09:55:50.071767 1739 leader_election.cc:302] T 209567a1c4a24d029c52b6bb3e940945 P 7bda7656bed9450db86559665a6cc3fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7bda7656bed9450db86559665a6cc3fd, e2fbec6f2e574d4bbfcf0fdf050449c5; no voters:
I20260924 09:55:50.072052 2020 raft_consensus.cc:2781] T 209567a1c4a24d029c52b6bb3e940945 P 7bda7656bed9450db86559665a6cc3fd [term 1 FOLLOWER]: Leader election won for term 1
I20260924 09:55:50.072427 2020 raft_consensus.cc:686] T 209567a1c4a24d029c52b6bb3e940945 P 7bda7656bed9450db86559665a6cc3fd [term 1 LEADER]: Becoming Leader. State: Replica: 7bda7656bed9450db86559665a6cc3fd, State: Running, Role: LEADER
I20260924 09:55:50.072875 2020 consensus_queue.cc:237] T 209567a1c4a24d029c52b6bb3e940945 P 7bda7656bed9450db86559665a6cc3fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50155961724d43b585d6e12c12bab09a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44123 } } peers { permanent_uuid: "7bda7656bed9450db86559665a6cc3fd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34007 } } peers { permanent_uuid: "e2fbec6f2e574d4bbfcf0fdf050449c5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35531 } }
I20260924 09:55:50.077288 1511 catalog_manager.cc:5495] T 209567a1c4a24d029c52b6bb3e940945 P 7bda7656bed9450db86559665a6cc3fd reported cstate change: term changed from 0 to 1, leader changed from <none> to 7bda7656bed9450db86559665a6cc3fd (127.0.97.2). New cstate: current_term: 1 leader_uuid: "7bda7656bed9450db86559665a6cc3fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50155961724d43b585d6e12c12bab09a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44123 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7bda7656bed9450db86559665a6cc3fd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e2fbec6f2e574d4bbfcf0fdf050449c5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35531 } health_report { overall_health: UNKNOWN } } }
I20260924 09:55:50.110741 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:55:50.105585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59368 (local address 127.0.97.2:34007)
0924 09:55:50.105769 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0924 09:55:50.105776 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 09:55:50.105864 (+ 88us) server_negotiation.cc:407] Connection header received
0924 09:55:50.106165 (+ 301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:55:50.106169 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:55:50.106254 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:55:50.106373 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 09:55:50.106712 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:55:50.107437 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:55:50.108701 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:55:50.109092 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:55:50.109207 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:55:50.109672 (+ 465us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 09:55:50.110180 (+ 508us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 09:55:50.110303 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0924 09:55:50.110419 (+ 116us) server_negotiation.cc:299] Negotiation successful
0924 09:55:50.110497 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 09:55:50.118598 1672 raft_consensus.cc:1261] T 209567a1c4a24d029c52b6bb3e940945 P 50155961724d43b585d6e12c12bab09a [term 1 FOLLOWER]: Refusing update from remote peer 7bda7656bed9450db86559665a6cc3fd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 09:55:50.118597 1951 raft_consensus.cc:1261] T 209567a1c4a24d029c52b6bb3e940945 P e2fbec6f2e574d4bbfcf0fdf050449c5 [term 1 FOLLOWER]: Refusing update from remote peer 7bda7656bed9450db86559665a6cc3fd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 09:55:50.119501 2020 consensus_queue.cc:1035] T 209567a1c4a24d029c52b6bb3e940945 P 7bda7656bed9450db86559665a6cc3fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "50155961724d43b585d6e12c12bab09a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 09:55:50.119760 2026 consensus_queue.cc:1035] T 209567a1c4a24d029c52b6bb3e940945 P 7bda7656bed9450db86559665a6cc3fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2fbec6f2e574d4bbfcf0fdf050449c5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 09:55:50.149652 1512 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57838:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 09:55:50.149869 1512 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57838:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 09:55:50.153760 1512 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9e23454a67c54530a74ec880bbe721d4: Sending DeleteTablet for 3 replicas of tablet 209567a1c4a24d029c52b6bb3e940945
I20260924 09:55:50.154892 1787 tablet_service.cc:1514] Processing DeleteTablet for tablet 209567a1c4a24d029c52b6bb3e940945 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 09:55:50 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:59360
I20260924 09:55:50.154881 1648 tablet_service.cc:1514] Processing DeleteTablet for tablet 209567a1c4a24d029c52b6bb3e940945 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 09:55:50 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:57902
I20260924 09:55:50.155076 1926 tablet_service.cc:1514] Processing DeleteTablet for tablet 209567a1c4a24d029c52b6bb3e940945 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 09:55:50 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:39216
I20260924 09:55:50.157126 1512 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57838:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 09:55:50.157796 1512 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 09:55:50.165195 1647 tablet_service.cc:1467] Processing CreateTablet for tablet 33524144dfbd4a50bdb4cb485de943ad (DEFAULT_TABLE table=test-table [id=93782efcaa0943eb801a23c7c454b312]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 09:55:50.165831 1647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33524144dfbd4a50bdb4cb485de943ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 09:55:50.168130 1786 tablet_service.cc:1467] Processing CreateTablet for tablet 33524144dfbd4a50bdb4cb485de943ad (DEFAULT_TABLE table=test-table [id=93782efcaa0943eb801a23c7c454b312]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 09:55:50.168637 1786 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33524144dfbd4a50bdb4cb485de943ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 09:55:50.168697 1925 tablet_service.cc:1467] Processing CreateTablet for tablet 33524144dfbd4a50bdb4cb485de943ad (DEFAULT_TABLE table=test-table [id=93782efcaa0943eb801a23c7c454b312]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 09:55:50.169219 1925 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33524144dfbd4a50bdb4cb485de943ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 09:55:50.170282 2039 tablet_replica.cc:330] T 209567a1c4a24d029c52b6bb3e940945 P 50155961724d43b585d6e12c12bab09a: stopping tablet replica
I20260924 09:55:50.170719 2039 raft_consensus.cc:2229] T 209567a1c4a24d029c52b6bb3e940945 P 50155961724d43b585d6e12c12bab09a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 09:55:50.171197 2039 raft_consensus.cc:2258] T 209567a1c4a24d029c52b6bb3e940945 P 50155961724d43b585d6e12c12bab09a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 09:55:50.173926 2039 ts_tablet_manager.cc:1905] T 209567a1c4a24d029c52b6bb3e940945 P 50155961724d43b585d6e12c12bab09a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 09:55:50.175135 2018 tablet_bootstrap.cc:492] T 33524144dfbd4a50bdb4cb485de943ad P e2fbec6f2e574d4bbfcf0fdf050449c5: Bootstrap starting.
I20260924 09:55:50.175943 2016 tablet_bootstrap.cc:492] T 33524144dfbd4a50bdb4cb485de943ad P 50155961724d43b585d6e12c12bab09a: Bootstrap starting.
I20260924 09:55:50.176299 2015 tablet_bootstrap.cc:492] T 33524144dfbd4a50bdb4cb485de943ad P 7bda7656bed9450db86559665a6cc3fd: Bootstrap starting.
I20260924 09:55:50.177098 2018 tablet_bootstrap.cc:654] T 33524144dfbd4a50bdb4cb485de943ad P e2fbec6f2e574d4bbfcf0fdf050449c5: Neither blocks nor log segments found. Creating new log.
I20260924 09:55:50.177476 2016 tablet_bootstrap.cc:654] T 33524144dfbd4a50bdb4cb485de943ad P 50155961724d43b585d6e12c12bab09a: Neither blocks nor log segments found. Creating new log.
I20260924 09:55:50.177773 2015 tablet_bootstrap.cc:654] T 33524144dfbd4a50bdb4cb485de943ad P 7bda7656bed9450db86559665a6cc3fd: Neither blocks nor log segments found. Creating new log.
I20260924 09:55:50.183089 2016 tablet_bootstrap.cc:492] T 33524144dfbd4a50bdb4cb485de943ad P 50155961724d43b585d6e12c12bab09a: No bootstrap required, opened a new log
I20260924 09:55:50.183331 2016 ts_tablet_manager.cc:1397] T 33524144dfbd4a50bdb4cb485de943ad P 50155961724d43b585d6e12c12bab09a: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260924 09:55:50.185300 2040 tablet_replica.cc:330] T 209567a1c4a24d029c52b6bb3e940945 P 7bda7656bed9450db86559665a6cc3fd: stopping tablet replica
I20260924 09:55:50.185747 2040 raft_consensus.cc:2229] T 209567a1c4a24d029c52b6bb3e940945 P 7bda7656bed9450db86559665a6cc3fd [term 1 LEADER]: Raft consensus shutting down.
I20260924 09:55:50.186089 2016 raft_consensus.cc:348] T 33524144dfbd4a50bdb4cb485de943ad P 50155961724d43b585d6e12c12bab09a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50155961724d43b585d6e12c12bab09a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44123 } } peers { permanent_uuid: "7bda7656bed9450db86559665a6cc3fd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34007 } } peers { permanent_uuid: "e2fbec6f2e574d4bbfcf0fdf050449c5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35531 } }
I20260924 09:55:50.186273 2041 tablet_replica.cc:330] T 209567a1c4a24d029c52b6bb3e940945 P e2fbec6f2e574d4bbfcf0fdf050449c5: stopping tablet replica
I20260924 09:55:50.186395 2016 raft_consensus.cc:374] T 33524144dfbd4a50bdb4cb485de943ad P 50155961724d43b585d6e12c12bab09a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 09:55:50.186532 2016 raft_consensus.cc:729] T 33524144dfbd4a50bdb4cb485de943ad P 50155961724d43b585d6e12c12bab09a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50155961724d43b585d6e12c12bab09a, State: Initialized, Role: FOLLOWER
I20260924 09:55:50.186683 2041 raft_consensus.cc:2229] T 209567a1c4a24d029c52b6bb3e940945 P e2fbec6f2e574d4bbfcf0fdf050449c5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 09:55:50.187014 2039 ts_tablet_manager.cc:1918] T 209567a1c4a24d029c52b6bb3e940945 P 50155961724d43b585d6e12c12bab09a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 09:55:50.186939 2016 consensus_queue.cc:260] T 33524144dfbd4a50bdb4cb485de943ad P 50155961724d43b585d6e12c12bab09a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50155961724d43b585d6e12c12bab09a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44123 } } peers { permanent_uuid: "7bda7656bed9450db86559665a6cc3fd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34007 } } peers { permanent_uuid: "e2fbec6f2e574d4bbfcf0fdf050449c5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35531 } }
I20260924 09:55:50.187157 2041 raft_consensus.cc:2258] T 209567a1c4a24d029c52b6bb3e940945 P e2fbec6f2e574d4bbfcf0fdf050449c5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 09:55:50.188092 2040 raft_consensus.cc:2258] T 209567a1c4a24d029c52b6bb3e940945 P 7bda7656bed9450db86559665a6cc3fd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 09:55:50.189357 2016 ts_tablet_manager.cc:1428] T 33524144dfbd4a50bdb4cb485de943ad P 50155961724d43b585d6e12c12bab09a: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20260924 09:55:50.189821 2040 ts_tablet_manager.cc:1905] T 209567a1c4a24d029c52b6bb3e940945 P 7bda7656bed9450db86559665a6cc3fd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 09:55:50.191332 2041 ts_tablet_manager.cc:1905] T 209567a1c4a24d029c52b6bb3e940945 P e2fbec6f2e574d4bbfcf0fdf050449c5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 09:55:50.193112 2018 tablet_bootstrap.cc:492] T 33524144dfbd4a50bdb4cb485de943ad P e2fbec6f2e574d4bbfcf0fdf050449c5: No bootstrap required, opened a new log
I20260924 09:55:50.193259 2018 ts_tablet_manager.cc:1397] T 33524144dfbd4a50bdb4cb485de943ad P e2fbec6f2e574d4bbfcf0fdf050449c5: Time spent bootstrapping tablet: real 0.018s user 0.003s sys 0.007s
I20260924 09:55:50.194103 2018 raft_consensus.cc:348] T 33524144dfbd4a50bdb4cb485de943ad P e2fbec6f2e574d4bbfcf0fdf050449c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50155961724d43b585d6e12c12bab09a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44123 } } peers { permanent_uuid: "7bda7656bed9450db86559665a6cc3fd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34007 } } peers { permanent_uuid: "e2fbec6f2e574d4bbfcf0fdf050449c5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35531 } }
I20260924 09:55:50.194334 2018 raft_consensus.cc:374] T 33524144dfbd4a50bdb4cb485de943ad P e2fbec6f2e574d4bbfcf0fdf050449c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 09:55:50.194437 2018 raft_consensus.cc:729] T 33524144dfbd4a50bdb4cb485de943ad P e2fbec6f2e574d4bbfcf0fdf050449c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2fbec6f2e574d4bbfcf0fdf050449c5, State: Initialized, Role: FOLLOWER
I20260924 09:55:50.194751 2018 consensus_queue.cc:260] T 33524144dfbd4a50bdb4cb485de943ad P e2fbec6f2e574d4bbfcf0fdf050449c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50155961724d43b585d6e12c12bab09a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44123 } } peers { permanent_uuid: "7bda7656bed9450db86559665a6cc3fd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34007 } } peers { permanent_uuid: "e2fbec6f2e574d4bbfcf0fdf050449c5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35531 } }
I20260924 09:55:50.196915 2040 ts_tablet_manager.cc:1918] T 209567a1c4a24d029c52b6bb3e940945 P 7bda7656bed9450db86559665a6cc3fd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 09:55:50.197077 2040 log.cc:1196] T 209567a1c4a24d029c52b6bb3e940945 P 7bda7656bed9450db86559665a6cc3fd: Deleting WAL directory at /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790243735634835-388-0/minicluster-data/ts-1/wal/wals/209567a1c4a24d029c52b6bb3e940945
I20260924 09:55:50.197453 2018 ts_tablet_manager.cc:1428] T 33524144dfbd4a50bdb4cb485de943ad P e2fbec6f2e574d4bbfcf0fdf050449c5: Time spent starting tablet: real 0.004s user 0.002s sys 0.001s
I20260924 09:55:50.197544 2040 ts_tablet_manager.cc:1939] T 209567a1c4a24d029c52b6bb3e940945 P 7bda7656bed9450db86559665a6cc3fd: Deleting consensus metadata
I20260924 09:55:50.199163 1496 catalog_manager.cc:4841] TS 7bda7656bed9450db86559665a6cc3fd (127.0.97.2:34007): tablet 209567a1c4a24d029c52b6bb3e940945 (table test-table [id=88b1bae35de34ce5a913630103050ecd]) successfully deleted
I20260924 09:55:50.199401 2041 ts_tablet_manager.cc:1918] T 209567a1c4a24d029c52b6bb3e940945 P e2fbec6f2e574d4bbfcf0fdf050449c5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 09:55:50.187173 2039 log.cc:1196] T 209567a1c4a24d029c52b6bb3e940945 P 50155961724d43b585d6e12c12bab09a: Deleting WAL directory at /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790243735634835-388-0/minicluster-data/ts-0/wal/wals/209567a1c4a24d029c52b6bb3e940945
I20260924 09:55:50.199676 2041 log.cc:1196] T 209567a1c4a24d029c52b6bb3e940945 P e2fbec6f2e574d4bbfcf0fdf050449c5: Deleting WAL directory at /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790243735634835-388-0/minicluster-data/ts-2/wal/wals/209567a1c4a24d029c52b6bb3e940945
I20260924 09:55:50.200009 2039 ts_tablet_manager.cc:1939] T 209567a1c4a24d029c52b6bb3e940945 P 50155961724d43b585d6e12c12bab09a: Deleting consensus metadata
I20260924 09:55:50.200129 2041 ts_tablet_manager.cc:1939] T 209567a1c4a24d029c52b6bb3e940945 P e2fbec6f2e574d4bbfcf0fdf050449c5: Deleting consensus metadata
I20260924 09:55:50.201424 1497 catalog_manager.cc:4841] TS e2fbec6f2e574d4bbfcf0fdf050449c5 (127.0.97.3:35531): tablet 209567a1c4a24d029c52b6bb3e940945 (table test-table [id=88b1bae35de34ce5a913630103050ecd]) successfully deleted
I20260924 09:55:50.201689 1499 catalog_manager.cc:4841] TS 50155961724d43b585d6e12c12bab09a (127.0.97.1:44123): tablet 209567a1c4a24d029c52b6bb3e940945 (table test-table [id=88b1bae35de34ce5a913630103050ecd]) successfully deleted
I20260924 09:55:50.202512 2015 tablet_bootstrap.cc:492] T 33524144dfbd4a50bdb4cb485de943ad P 7bda7656bed9450db86559665a6cc3fd: No bootstrap required, opened a new log
I20260924 09:55:50.202661 2015 ts_tablet_manager.cc:1397] T 33524144dfbd4a50bdb4cb485de943ad P 7bda7656bed9450db86559665a6cc3fd: Time spent bootstrapping tablet: real 0.027s user 0.005s sys 0.007s
I20260924 09:55:50.203502 2015 raft_consensus.cc:348] T 33524144dfbd4a50bdb4cb485de943ad P 7bda7656bed9450db86559665a6cc3fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50155961724d43b585d6e12c12bab09a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44123 } } peers { permanent_uuid: "7bda7656bed9450db86559665a6cc3fd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34007 } } peers { permanent_uuid: "e2fbec6f2e574d4bbfcf0fdf050449c5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35531 } }
I20260924 09:55:50.203712 2015 raft_consensus.cc:374] T 33524144dfbd4a50bdb4cb485de943ad P 7bda7656bed9450db86559665a6cc3fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 09:55:50.203823 2015 raft_consensus.cc:729] T 33524144dfbd4a50bdb4cb485de943ad P 7bda7656bed9450db86559665a6cc3fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7bda7656bed9450db86559665a6cc3fd, State: Initialized, Role: FOLLOWER
I20260924 09:55:50.204075 2015 consensus_queue.cc:260] T 33524144dfbd4a50bdb4cb485de943ad P 7bda7656bed9450db86559665a6cc3fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50155961724d43b585d6e12c12bab09a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44123 } } peers { permanent_uuid: "7bda7656bed9450db86559665a6cc3fd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34007 } } peers { permanent_uuid: "e2fbec6f2e574d4bbfcf0fdf050449c5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35531 } }
I20260924 09:55:50.204514 2015 ts_tablet_manager.cc:1428] T 33524144dfbd4a50bdb4cb485de943ad P 7bda7656bed9450db86559665a6cc3fd: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
W20260924 09:55:50.383663 1997 tablet.cc:2369] T 33524144dfbd4a50bdb4cb485de943ad P e2fbec6f2e574d4bbfcf0fdf050449c5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 09:55:50.405150 2023 raft_consensus.cc:482] T 33524144dfbd4a50bdb4cb485de943ad P e2fbec6f2e574d4bbfcf0fdf050449c5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 09:55:50.405354 2023 raft_consensus.cc:504] T 33524144dfbd4a50bdb4cb485de943ad P e2fbec6f2e574d4bbfcf0fdf050449c5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50155961724d43b585d6e12c12bab09a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44123 } } peers { permanent_uuid: "7bda7656bed9450db86559665a6cc3fd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34007 } } peers { permanent_uuid: "e2fbec6f2e574d4bbfcf0fdf050449c5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35531 } }
I20260924 09:55:50.406782 2023 leader_election.cc:288] T 33524144dfbd4a50bdb4cb485de943ad P e2fbec6f2e574d4bbfcf0fdf050449c5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 50155961724d43b585d6e12c12bab09a (127.0.97.1:44123), 7bda7656bed9450db86559665a6cc3fd (127.0.97.2:34007)
W20260924 09:55:50.407773 1858 tablet.cc:2369] T 33524144dfbd4a50bdb4cb485de943ad P 7bda7656bed9450db86559665a6cc3fd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 09:55:50.414994 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:55:50.407498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:55033 (local address 127.0.97.1:44123)
0924 09:55:50.407683 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0924 09:55:50.407689 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 09:55:50.408224 (+ 535us) server_negotiation.cc:407] Connection header received
0924 09:55:50.408483 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:55:50.408489 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:55:50.408554 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:55:50.408621 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 09:55:50.409188 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:55:50.410272 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:55:50.412655 (+ 2383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:55:50.413856 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:55:50.413965 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:55:50.414065 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0924 09:55:50.414583 (+ 518us) server_negotiation.cc:299] Negotiation successful
0924 09:55:50.414724 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 09:55:50.415948 2045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:55:50.407034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44123 (local address 127.0.97.3:55033)
0924 09:55:50.408100 (+ 1066us) negotiation.cc:107] Waiting for socket to connect
0924 09:55:50.408120 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 09:55:50.408330 (+ 210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:55:50.408784 (+ 454us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:55:50.408788 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:55:50.408810 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 09:55:50.409012 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:55:50.409021 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:55:50.410435 (+ 1414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:55:50.410440 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:55:50.412466 (+ 2026us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:55:50.412479 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:55:50.414217 (+ 1738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:55:50.414222 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:55:50.414401 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:55:50.414434 (+ 33us) client_negotiation.cc:769] Sending connection context
0924 09:55:50.415460 (+ 1026us) client_negotiation.cc:240] Negotiation successful
0924 09:55:50.415612 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":958,"mutex_wait_us":3,"thread_start_us":780,"threads_started":1}
I20260924 09:55:50.415994 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:55:50.407471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:51801 (local address 127.0.97.2:34007)
0924 09:55:50.407680 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0924 09:55:50.407687 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 09:55:50.408471 (+ 784us) server_negotiation.cc:407] Connection header received
0924 09:55:50.408715 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:55:50.408721 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:55:50.408793 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:55:50.408852 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 09:55:50.409505 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:55:50.411093 (+ 1588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:55:50.413414 (+ 2321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:55:50.414709 (+ 1295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:55:50.414834 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:55:50.414926 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0924 09:55:50.415528 (+ 602us) server_negotiation.cc:299] Negotiation successful
0924 09:55:50.415675 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260924 09:55:50.416633 2046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:55:50.407141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:34007 (local address 127.0.97.3:51801)
0924 09:55:50.408380 (+ 1239us) negotiation.cc:107] Waiting for socket to connect
0924 09:55:50.408397 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 09:55:50.408559 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:55:50.409039 (+ 480us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:55:50.409043 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:55:50.409070 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 09:55:50.409344 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:55:50.409354 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:55:50.411286 (+ 1932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:55:50.411291 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:55:50.413219 (+ 1928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:55:50.413233 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:55:50.415151 (+ 1918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:55:50.415158 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:55:50.415335 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:55:50.415368 (+ 33us) client_negotiation.cc:769] Sending connection context
0924 09:55:50.416227 (+ 859us) client_negotiation.cc:240] Negotiation successful
0924 09:55:50.416363 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1132,"thread_start_us":987,"threads_started":1}
I20260924 09:55:50.416867 1672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33524144dfbd4a50bdb4cb485de943ad" candidate_uuid: "e2fbec6f2e574d4bbfcf0fdf050449c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50155961724d43b585d6e12c12bab09a" is_pre_election: true
I20260924 09:55:50.417111 1672 raft_consensus.cc:2454] T 33524144dfbd4a50bdb4cb485de943ad P 50155961724d43b585d6e12c12bab09a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2fbec6f2e574d4bbfcf0fdf050449c5 in term 0.
I20260924 09:55:50.417605 1880 leader_election.cc:302] T 33524144dfbd4a50bdb4cb485de943ad P e2fbec6f2e574d4bbfcf0fdf050449c5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 50155961724d43b585d6e12c12bab09a, e2fbec6f2e574d4bbfcf0fdf050449c5; no voters:
I20260924 09:55:50.418224 2023 raft_consensus.cc:2781] T 33524144dfbd4a50bdb4cb485de943ad P e2fbec6f2e574d4bbfcf0fdf050449c5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 09:55:50.418339 2023 raft_consensus.cc:482] T 33524144dfbd4a50bdb4cb485de943ad P e2fbec6f2e574d4bbfcf0fdf050449c5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 09:55:50.418444 2023 raft_consensus.cc:3037] T 33524144dfbd4a50bdb4cb485de943ad P e2fbec6f2e574d4bbfcf0fdf050449c5 [term 0 FOLLOWER]: Advancing to term 1
I20260924 09:55:50.418308 1812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33524144dfbd4a50bdb4cb485de943ad" candidate_uuid: "e2fbec6f2e574d4bbfcf0fdf050449c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7bda7656bed9450db86559665a6cc3fd" is_pre_election: true
I20260924 09:55:50.418767 1812 raft_consensus.cc:2454] T 33524144dfbd4a50bdb4cb485de943ad P 7bda7656bed9450db86559665a6cc3fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2fbec6f2e574d4bbfcf0fdf050449c5 in term 0.
I20260924 09:55:50.421242 2023 raft_consensus.cc:504] T 33524144dfbd4a50bdb4cb485de943ad P e2fbec6f2e574d4bbfcf0fdf050449c5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50155961724d43b585d6e12c12bab09a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44123 } } peers { permanent_uuid: "7bda7656bed9450db86559665a6cc3fd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34007 } } peers { permanent_uuid: "e2fbec6f2e574d4bbfcf0fdf050449c5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35531 } }
I20260924 09:55:50.421782 2023 leader_election.cc:288] T 33524144dfbd4a50bdb4cb485de943ad P e2fbec6f2e574d4bbfcf0fdf050449c5 [CANDIDATE]: Term 1 election: Requested vote from peers 50155961724d43b585d6e12c12bab09a (127.0.97.1:44123), 7bda7656bed9450db86559665a6cc3fd (127.0.97.2:34007)
I20260924 09:55:50.422184 1672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33524144dfbd4a50bdb4cb485de943ad" candidate_uuid: "e2fbec6f2e574d4bbfcf0fdf050449c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50155961724d43b585d6e12c12bab09a"
I20260924 09:55:50.422384 1672 raft_consensus.cc:3037] T 33524144dfbd4a50bdb4cb485de943ad P 50155961724d43b585d6e12c12bab09a [term 0 FOLLOWER]: Advancing to term 1
I20260924 09:55:50.422417 1812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33524144dfbd4a50bdb4cb485de943ad" candidate_uuid: "e2fbec6f2e574d4bbfcf0fdf050449c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7bda7656bed9450db86559665a6cc3fd"
I20260924 09:55:50.422590 1812 raft_consensus.cc:3037] T 33524144dfbd4a50bdb4cb485de943ad P 7bda7656bed9450db86559665a6cc3fd [term 0 FOLLOWER]: Advancing to term 1
W20260924 09:55:50.424602 1719 tablet.cc:2369] T 33524144dfbd4a50bdb4cb485de943ad P 50155961724d43b585d6e12c12bab09a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 09:55:50.425122 1672 raft_consensus.cc:2454] T 33524144dfbd4a50bdb4cb485de943ad P 50155961724d43b585d6e12c12bab09a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2fbec6f2e574d4bbfcf0fdf050449c5 in term 1.
I20260924 09:55:50.425525 1880 leader_election.cc:302] T 33524144dfbd4a50bdb4cb485de943ad P e2fbec6f2e574d4bbfcf0fdf050449c5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 50155961724d43b585d6e12c12bab09a, e2fbec6f2e574d4bbfcf0fdf050449c5; no voters:
I20260924 09:55:50.425673 1812 raft_consensus.cc:2454] T 33524144dfbd4a50bdb4cb485de943ad P 7bda7656bed9450db86559665a6cc3fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2fbec6f2e574d4bbfcf0fdf050449c5 in term 1.
I20260924 09:55:50.425865 2023 raft_consensus.cc:2781] T 33524144dfbd4a50bdb4cb485de943ad P e2fbec6f2e574d4bbfcf0fdf050449c5 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 09:55:50.426465 2023 raft_consensus.cc:686] T 33524144dfbd4a50bdb4cb485de943ad P e2fbec6f2e574d4bbfcf0fdf050449c5 [term 1 LEADER]: Becoming Leader. State: Replica: e2fbec6f2e574d4bbfcf0fdf050449c5, State: Running, Role: LEADER
I20260924 09:55:50.426999 2023 consensus_queue.cc:237] T 33524144dfbd4a50bdb4cb485de943ad P e2fbec6f2e574d4bbfcf0fdf050449c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50155961724d43b585d6e12c12bab09a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44123 } } peers { permanent_uuid: "7bda7656bed9450db86559665a6cc3fd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34007 } } peers { permanent_uuid: "e2fbec6f2e574d4bbfcf0fdf050449c5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35531 } }
I20260924 09:55:50.430160 1511 catalog_manager.cc:5495] T 33524144dfbd4a50bdb4cb485de943ad P e2fbec6f2e574d4bbfcf0fdf050449c5 reported cstate change: term changed from 0 to 1, leader changed from <none> to e2fbec6f2e574d4bbfcf0fdf050449c5 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "e2fbec6f2e574d4bbfcf0fdf050449c5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50155961724d43b585d6e12c12bab09a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44123 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7bda7656bed9450db86559665a6cc3fd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2fbec6f2e574d4bbfcf0fdf050449c5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35531 } health_report { overall_health: HEALTHY } } }
Sep 24 09:55:50 dist-test-slave-w8d2 krb5kdc[1467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243748, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 09:55:50.519601 2054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:55:50.506236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42407 (local address 127.0.0.1:57852)
0924 09:55:50.506849 (+ 613us) negotiation.cc:107] Waiting for socket to connect
0924 09:55:50.506871 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 09:55:50.507036 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:55:50.507436 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:55:50.507441 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:55:50.508507 (+ 1066us) client_negotiation.cc:189] Negotiated authn=SASL
0924 09:55:50.508680 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:55:50.508689 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:55:50.509695 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:55:50.509700 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:55:50.510589 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:55:50.510598 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:55:50.511267 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:55:50.511273 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:55:50.511449 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:55:50.512107 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 09:55:50.512123 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 09:55:50.514482 (+ 2359us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 09:55:50.517255 (+ 2773us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 09:55:50.517267 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 09:55:50.517275 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0924 09:55:50.517669 (+ 394us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 09:55:50.517979 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 09:55:50.517984 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 09:55:50.517988 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 09:55:50.518106 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 09:55:50.518562 (+ 456us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 09:55:50.518571 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 09:55:50.518753 (+ 182us) client_negotiation.cc:769] Sending connection context
0924 09:55:50.518904 (+ 151us) client_negotiation.cc:240] Negotiation successful
0924 09:55:50.519173 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":319,"threads_started":1}
I20260924 09:55:50.519675 2055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:55:50.506351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57852 (local address 127.0.97.62:42407)
0924 09:55:50.507156 (+ 805us) server_negotiation.cc:206] Beginning negotiation
0924 09:55:50.507161 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 09:55:50.507179 (+ 18us) server_negotiation.cc:407] Connection header received
0924 09:55:50.507219 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:55:50.507222 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:55:50.507304 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:55:50.507389 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:55:50.508815 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:55:50.509568 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:55:50.510720 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:55:50.511101 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:55:50.511201 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:55:50.514641 (+ 3440us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:55:50.514646 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:55:50.514649 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:55:50.514675 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:55:50.517128 (+ 2453us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:55:50.517768 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:55:50.517772 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:55:50.517776 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:55:50.517888 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:55:50.518192 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:55:50.518197 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:55:50.518207 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:55:50.518451 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:55:50.518569 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0924 09:55:50.518990 (+ 421us) server_negotiation.cc:299] Negotiation successful
0924 09:55:50.519337 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":130,"server-negotiator.queue_time_us":682,"thread_start_us":545,"threads_started":1}
I20260924 09:55:50.524978 1511 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:57852:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260924 09:55:50.525976 1511 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 09:55:50.532501 1647 tablet_service.cc:1467] Processing CreateTablet for tablet 892f9fad56544b6899146ea3ca42f0f2 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d8de3ffdbc6f46749dd7967d2188c869]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 09:55:50.532699 1926 tablet_service.cc:1467] Processing CreateTablet for tablet 892f9fad56544b6899146ea3ca42f0f2 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d8de3ffdbc6f46749dd7967d2188c869]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 09:55:50.532809 1787 tablet_service.cc:1467] Processing CreateTablet for tablet 892f9fad56544b6899146ea3ca42f0f2 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d8de3ffdbc6f46749dd7967d2188c869]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 09:55:50.533039 1647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 892f9fad56544b6899146ea3ca42f0f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 09:55:50.533185 1926 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 892f9fad56544b6899146ea3ca42f0f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 09:55:50.533347 1787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 892f9fad56544b6899146ea3ca42f0f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 09:55:50.539937 2016 tablet_bootstrap.cc:492] T 892f9fad56544b6899146ea3ca42f0f2 P 50155961724d43b585d6e12c12bab09a: Bootstrap starting.
I20260924 09:55:50.539978 2018 tablet_bootstrap.cc:492] T 892f9fad56544b6899146ea3ca42f0f2 P e2fbec6f2e574d4bbfcf0fdf050449c5: Bootstrap starting.
I20260924 09:55:50.541745 2016 tablet_bootstrap.cc:654] T 892f9fad56544b6899146ea3ca42f0f2 P 50155961724d43b585d6e12c12bab09a: Neither blocks nor log segments found. Creating new log.
I20260924 09:55:50.541841 2018 tablet_bootstrap.cc:654] T 892f9fad56544b6899146ea3ca42f0f2 P e2fbec6f2e574d4bbfcf0fdf050449c5: Neither blocks nor log segments found. Creating new log.
I20260924 09:55:50.543665 2016 tablet_bootstrap.cc:492] T 892f9fad56544b6899146ea3ca42f0f2 P 50155961724d43b585d6e12c12bab09a: No bootstrap required, opened a new log
I20260924 09:55:50.543812 2016 ts_tablet_manager.cc:1397] T 892f9fad56544b6899146ea3ca42f0f2 P 50155961724d43b585d6e12c12bab09a: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.001s
I20260924 09:55:50.544363 2016 raft_consensus.cc:348] T 892f9fad56544b6899146ea3ca42f0f2 P 50155961724d43b585d6e12c12bab09a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50155961724d43b585d6e12c12bab09a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44123 } } peers { permanent_uuid: "7bda7656bed9450db86559665a6cc3fd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34007 } } peers { permanent_uuid: "e2fbec6f2e574d4bbfcf0fdf050449c5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35531 } }
I20260924 09:55:50.544548 2016 raft_consensus.cc:374] T 892f9fad56544b6899146ea3ca42f0f2 P 50155961724d43b585d6e12c12bab09a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 09:55:50.544615 2016 raft_consensus.cc:729] T 892f9fad56544b6899146ea3ca42f0f2 P 50155961724d43b585d6e12c12bab09a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50155961724d43b585d6e12c12bab09a, State: Initialized, Role: FOLLOWER
I20260924 09:55:50.544837 2016 consensus_queue.cc:260] T 892f9fad56544b6899146ea3ca42f0f2 P 50155961724d43b585d6e12c12bab09a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50155961724d43b585d6e12c12bab09a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44123 } } peers { permanent_uuid: "7bda7656bed9450db86559665a6cc3fd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34007 } } peers { permanent_uuid: "e2fbec6f2e574d4bbfcf0fdf050449c5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35531 } }
I20260924 09:55:50.545233 2022 tablet_replica.cc:439] T 892f9fad56544b6899146ea3ca42f0f2 P 50155961724d43b585d6e12c12bab09a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50155961724d43b585d6e12c12bab09a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44123 } } peers { permanent_uuid: "7bda7656bed9450db86559665a6cc3fd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34007 } } peers { permanent_uuid: "e2fbec6f2e574d4bbfcf0fdf050449c5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35531 } } }
I20260924 09:55:50.545578 2022 tablet_replica.cc:442] T 892f9fad56544b6899146ea3ca42f0f2 P 50155961724d43b585d6e12c12bab09a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 09:55:50.546366 2016 ts_tablet_manager.cc:1428] T 892f9fad56544b6899146ea3ca42f0f2 P 50155961724d43b585d6e12c12bab09a: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260924 09:55:50.547495 2015 tablet_bootstrap.cc:492] T 892f9fad56544b6899146ea3ca42f0f2 P 7bda7656bed9450db86559665a6cc3fd: Bootstrap starting.
I20260924 09:55:50.549579 2015 tablet_bootstrap.cc:654] T 892f9fad56544b6899146ea3ca42f0f2 P 7bda7656bed9450db86559665a6cc3fd: Neither blocks nor log segments found. Creating new log.
I20260924 09:55:50.552459 2018 tablet_bootstrap.cc:492] T 892f9fad56544b6899146ea3ca42f0f2 P e2fbec6f2e574d4bbfcf0fdf050449c5: No bootstrap required, opened a new log
I20260924 09:55:50.552639 2018 ts_tablet_manager.cc:1397] T 892f9fad56544b6899146ea3ca42f0f2 P e2fbec6f2e574d4bbfcf0fdf050449c5: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.004s
I20260924 09:55:50.552915 2015 tablet_bootstrap.cc:492] T 892f9fad56544b6899146ea3ca42f0f2 P 7bda7656bed9450db86559665a6cc3fd: No bootstrap required, opened a new log
I20260924 09:55:50.553085 2015 ts_tablet_manager.cc:1397] T 892f9fad56544b6899146ea3ca42f0f2 P 7bda7656bed9450db86559665a6cc3fd: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.001s
I20260924 09:55:50.553444 2018 raft_consensus.cc:348] T 892f9fad56544b6899146ea3ca42f0f2 P e2fbec6f2e574d4bbfcf0fdf050449c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50155961724d43b585d6e12c12bab09a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44123 } } peers { permanent_uuid: "7bda7656bed9450db86559665a6cc3fd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34007 } } peers { permanent_uuid: "e2fbec6f2e574d4bbfcf0fdf050449c5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35531 } }
I20260924 09:55:50.553683 2018 raft_consensus.cc:374] T 892f9fad56544b6899146ea3ca42f0f2 P e2fbec6f2e574d4bbfcf0fdf050449c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 09:55:50.553754 2018 raft_consensus.cc:729] T 892f9fad56544b6899146ea3ca42f0f2 P e2fbec6f2e574d4bbfcf0fdf050449c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2fbec6f2e574d4bbfcf0fdf050449c5, State: Initialized, Role: FOLLOWER
I20260924 09:55:50.553735 2015 raft_consensus.cc:348] T 892f9fad56544b6899146ea3ca42f0f2 P 7bda7656bed9450db86559665a6cc3fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50155961724d43b585d6e12c12bab09a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44123 } } peers { permanent_uuid: "7bda7656bed9450db86559665a6cc3fd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34007 } } peers { permanent_uuid: "e2fbec6f2e574d4bbfcf0fdf050449c5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35531 } }
I20260924 09:55:50.553958 2015 raft_consensus.cc:374] T 892f9fad56544b6899146ea3ca42f0f2 P 7bda7656bed9450db86559665a6cc3fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 09:55:50.554039 2015 raft_consensus.cc:729] T 892f9fad56544b6899146ea3ca42f0f2 P 7bda7656bed9450db86559665a6cc3fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7bda7656bed9450db86559665a6cc3fd, State: Initialized, Role: FOLLOWER
I20260924 09:55:50.553962 2018 consensus_queue.cc:260] T 892f9fad56544b6899146ea3ca42f0f2 P e2fbec6f2e574d4bbfcf0fdf050449c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50155961724d43b585d6e12c12bab09a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44123 } } peers { permanent_uuid: "7bda7656bed9450db86559665a6cc3fd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34007 } } peers { permanent_uuid: "e2fbec6f2e574d4bbfcf0fdf050449c5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35531 } }
I20260924 09:55:50.554308 2015 consensus_queue.cc:260] T 892f9fad56544b6899146ea3ca42f0f2 P 7bda7656bed9450db86559665a6cc3fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50155961724d43b585d6e12c12bab09a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44123 } } peers { permanent_uuid: "7bda7656bed9450db86559665a6cc3fd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34007 } } peers { permanent_uuid: "e2fbec6f2e574d4bbfcf0fdf050449c5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35531 } }
I20260924 09:55:50.554346 2023 tablet_replica.cc:439] T 892f9fad56544b6899146ea3ca42f0f2 P e2fbec6f2e574d4bbfcf0fdf050449c5: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50155961724d43b585d6e12c12bab09a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44123 } } peers { permanent_uuid: "7bda7656bed9450db86559665a6cc3fd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34007 } } peers { permanent_uuid: "e2fbec6f2e574d4bbfcf0fdf050449c5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35531 } } }
I20260924 09:55:50.554601 2023 tablet_replica.cc:442] T 892f9fad56544b6899146ea3ca42f0f2 P e2fbec6f2e574d4bbfcf0fdf050449c5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 09:55:50.554890 2020 tablet_replica.cc:439] T 892f9fad56544b6899146ea3ca42f0f2 P 7bda7656bed9450db86559665a6cc3fd: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50155961724d43b585d6e12c12bab09a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44123 } } peers { permanent_uuid: "7bda7656bed9450db86559665a6cc3fd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34007 } } peers { permanent_uuid: "e2fbec6f2e574d4bbfcf0fdf050449c5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35531 } } }
I20260924 09:55:50.555096 2015 ts_tablet_manager.cc:1428] T 892f9fad56544b6899146ea3ca42f0f2 P 7bda7656bed9450db86559665a6cc3fd: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260924 09:55:50.555138 2020 tablet_replica.cc:442] T 892f9fad56544b6899146ea3ca42f0f2 P 7bda7656bed9450db86559665a6cc3fd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 09:55:50.555974 2018 ts_tablet_manager.cc:1428] T 892f9fad56544b6899146ea3ca42f0f2 P e2fbec6f2e574d4bbfcf0fdf050449c5: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260924 09:55:50.618657 2020 raft_consensus.cc:482] T 892f9fad56544b6899146ea3ca42f0f2 P 7bda7656bed9450db86559665a6cc3fd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 09:55:50.618829 2020 raft_consensus.cc:504] T 892f9fad56544b6899146ea3ca42f0f2 P 7bda7656bed9450db86559665a6cc3fd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50155961724d43b585d6e12c12bab09a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44123 } } peers { permanent_uuid: "7bda7656bed9450db86559665a6cc3fd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34007 } } peers { permanent_uuid: "e2fbec6f2e574d4bbfcf0fdf050449c5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35531 } }
I20260924 09:55:50.619424 2020 leader_election.cc:288] T 892f9fad56544b6899146ea3ca42f0f2 P 7bda7656bed9450db86559665a6cc3fd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 50155961724d43b585d6e12c12bab09a (127.0.97.1:44123), e2fbec6f2e574d4bbfcf0fdf050449c5 (127.0.97.3:35531)
I20260924 09:55:50.619913 1672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "892f9fad56544b6899146ea3ca42f0f2" candidate_uuid: "7bda7656bed9450db86559665a6cc3fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50155961724d43b585d6e12c12bab09a" is_pre_election: true
I20260924 09:55:50.619927 1951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "892f9fad56544b6899146ea3ca42f0f2" candidate_uuid: "7bda7656bed9450db86559665a6cc3fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2fbec6f2e574d4bbfcf0fdf050449c5" is_pre_election: true
I20260924 09:55:50.620110 1951 raft_consensus.cc:2454] T 892f9fad56544b6899146ea3ca42f0f2 P e2fbec6f2e574d4bbfcf0fdf050449c5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7bda7656bed9450db86559665a6cc3fd in term 0.
I20260924 09:55:50.620141 1672 raft_consensus.cc:2454] T 892f9fad56544b6899146ea3ca42f0f2 P 50155961724d43b585d6e12c12bab09a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7bda7656bed9450db86559665a6cc3fd in term 0.
I20260924 09:55:50.620488 1739 leader_election.cc:302] T 892f9fad56544b6899146ea3ca42f0f2 P 7bda7656bed9450db86559665a6cc3fd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7bda7656bed9450db86559665a6cc3fd, e2fbec6f2e574d4bbfcf0fdf050449c5; no voters:
I20260924 09:55:50.620740 2020 raft_consensus.cc:2781] T 892f9fad56544b6899146ea3ca42f0f2 P 7bda7656bed9450db86559665a6cc3fd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 09:55:50.620852 2020 raft_consensus.cc:482] T 892f9fad56544b6899146ea3ca42f0f2 P 7bda7656bed9450db86559665a6cc3fd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 09:55:50.620941 2020 raft_consensus.cc:3037] T 892f9fad56544b6899146ea3ca42f0f2 P 7bda7656bed9450db86559665a6cc3fd [term 0 FOLLOWER]: Advancing to term 1
I20260924 09:55:50.623047 2020 raft_consensus.cc:504] T 892f9fad56544b6899146ea3ca42f0f2 P 7bda7656bed9450db86559665a6cc3fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50155961724d43b585d6e12c12bab09a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44123 } } peers { permanent_uuid: "7bda7656bed9450db86559665a6cc3fd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34007 } } peers { permanent_uuid: "e2fbec6f2e574d4bbfcf0fdf050449c5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35531 } }
I20260924 09:55:50.623554 2020 leader_election.cc:288] T 892f9fad56544b6899146ea3ca42f0f2 P 7bda7656bed9450db86559665a6cc3fd [CANDIDATE]: Term 1 election: Requested vote from peers 50155961724d43b585d6e12c12bab09a (127.0.97.1:44123), e2fbec6f2e574d4bbfcf0fdf050449c5 (127.0.97.3:35531)
I20260924 09:55:50.624019 1672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "892f9fad56544b6899146ea3ca42f0f2" candidate_uuid: "7bda7656bed9450db86559665a6cc3fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50155961724d43b585d6e12c12bab09a"
I20260924 09:55:50.624032 1951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "892f9fad56544b6899146ea3ca42f0f2" candidate_uuid: "7bda7656bed9450db86559665a6cc3fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2fbec6f2e574d4bbfcf0fdf050449c5"
I20260924 09:55:50.624195 1672 raft_consensus.cc:3037] T 892f9fad56544b6899146ea3ca42f0f2 P 50155961724d43b585d6e12c12bab09a [term 0 FOLLOWER]: Advancing to term 1
I20260924 09:55:50.624214 1951 raft_consensus.cc:3037] T 892f9fad56544b6899146ea3ca42f0f2 P e2fbec6f2e574d4bbfcf0fdf050449c5 [term 0 FOLLOWER]: Advancing to term 1
I20260924 09:55:50.626015 1672 raft_consensus.cc:2454] T 892f9fad56544b6899146ea3ca42f0f2 P 50155961724d43b585d6e12c12bab09a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7bda7656bed9450db86559665a6cc3fd in term 1.
I20260924 09:55:50.626129 1951 raft_consensus.cc:2454] T 892f9fad56544b6899146ea3ca42f0f2 P e2fbec6f2e574d4bbfcf0fdf050449c5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7bda7656bed9450db86559665a6cc3fd in term 1.
I20260924 09:55:50.626359 1741 leader_election.cc:302] T 892f9fad56544b6899146ea3ca42f0f2 P 7bda7656bed9450db86559665a6cc3fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 50155961724d43b585d6e12c12bab09a, 7bda7656bed9450db86559665a6cc3fd; no voters:
I20260924 09:55:50.626648 2020 raft_consensus.cc:2781] T 892f9fad56544b6899146ea3ca42f0f2 P 7bda7656bed9450db86559665a6cc3fd [term 1 FOLLOWER]: Leader election won for term 1
I20260924 09:55:50.626766 2020 raft_consensus.cc:686] T 892f9fad56544b6899146ea3ca42f0f2 P 7bda7656bed9450db86559665a6cc3fd [term 1 LEADER]: Becoming Leader. State: Replica: 7bda7656bed9450db86559665a6cc3fd, State: Running, Role: LEADER
I20260924 09:55:50.627009 2020 consensus_queue.cc:237] T 892f9fad56544b6899146ea3ca42f0f2 P 7bda7656bed9450db86559665a6cc3fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50155961724d43b585d6e12c12bab09a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44123 } } peers { permanent_uuid: "7bda7656bed9450db86559665a6cc3fd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34007 } } peers { permanent_uuid: "e2fbec6f2e574d4bbfcf0fdf050449c5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35531 } }
I20260924 09:55:50.628446 2034 tablet_replica.cc:439] T 892f9fad56544b6899146ea3ca42f0f2 P 7bda7656bed9450db86559665a6cc3fd: TxnStatusTablet state changed. Reason: New leader 7bda7656bed9450db86559665a6cc3fd. Latest consensus state: current_term: 1 leader_uuid: "7bda7656bed9450db86559665a6cc3fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50155961724d43b585d6e12c12bab09a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44123 } } peers { permanent_uuid: "7bda7656bed9450db86559665a6cc3fd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34007 } } peers { permanent_uuid: "e2fbec6f2e574d4bbfcf0fdf050449c5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35531 } } }
I20260924 09:55:50.628720 2034 tablet_replica.cc:442] T 892f9fad56544b6899146ea3ca42f0f2 P 7bda7656bed9450db86559665a6cc3fd: This TxnStatusTablet replica's current role is: LEADER
I20260924 09:55:50.629273 2060 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260924 09:55:50.629438 2060 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260924 09:55:50.629670 1511 catalog_manager.cc:5495] T 892f9fad56544b6899146ea3ca42f0f2 P 7bda7656bed9450db86559665a6cc3fd reported cstate change: term changed from 0 to 1, leader changed from <none> to 7bda7656bed9450db86559665a6cc3fd (127.0.97.2). New cstate: current_term: 1 leader_uuid: "7bda7656bed9450db86559665a6cc3fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50155961724d43b585d6e12c12bab09a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44123 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7bda7656bed9450db86559665a6cc3fd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e2fbec6f2e574d4bbfcf0fdf050449c5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35531 } health_report { overall_health: UNKNOWN } } }
I20260924 09:55:50.654582 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:55:50.649190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59376 (local address 127.0.97.2:34007)
0924 09:55:50.649378 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0924 09:55:50.649384 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 09:55:50.649499 (+ 115us) server_negotiation.cc:407] Connection header received
0924 09:55:50.649769 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:55:50.649773 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:55:50.649853 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:55:50.649930 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 09:55:50.650331 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:55:50.651127 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:55:50.653012 (+ 1885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:55:50.653439 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:55:50.653561 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:55:50.654020 (+ 459us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 09:55:50.654119 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 09:55:50.654185 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 09:55:50.654319 (+ 134us) server_negotiation.cc:299] Negotiation successful
0924 09:55:50.654378 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260924 09:55:50.654790 2054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:55:50.649169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:34007 (local address 127.0.0.1:59376)
0924 09:55:50.649412 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0924 09:55:50.649433 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 09:55:50.649587 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:55:50.649976 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:55:50.649982 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:55:50.649995 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 09:55:50.650205 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:55:50.650217 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:55:50.651289 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:55:50.651295 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:55:50.652880 (+ 1585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:55:50.652901 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:55:50.653581 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:55:50.653588 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:55:50.653792 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:55:50.653867 (+ 75us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 09:55:50.654205 (+ 338us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 09:55:50.654225 (+ 20us) client_negotiation.cc:769] Sending connection context
0924 09:55:50.654308 (+ 83us) client_negotiation.cc:240] Negotiation successful
0924 09:55:50.654460 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260924 09:55:50.658703 1951 raft_consensus.cc:1261] T 892f9fad56544b6899146ea3ca42f0f2 P e2fbec6f2e574d4bbfcf0fdf050449c5 [term 1 FOLLOWER]: Refusing update from remote peer 7bda7656bed9450db86559665a6cc3fd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 09:55:50.658878 1672 raft_consensus.cc:1261] T 892f9fad56544b6899146ea3ca42f0f2 P 50155961724d43b585d6e12c12bab09a [term 1 FOLLOWER]: Refusing update from remote peer 7bda7656bed9450db86559665a6cc3fd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 09:55:50.659334 2034 consensus_queue.cc:1035] T 892f9fad56544b6899146ea3ca42f0f2 P 7bda7656bed9450db86559665a6cc3fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2fbec6f2e574d4bbfcf0fdf050449c5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 09:55:50.659792 2020 consensus_queue.cc:1035] T 892f9fad56544b6899146ea3ca42f0f2 P 7bda7656bed9450db86559665a6cc3fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "50155961724d43b585d6e12c12bab09a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 09:55:50.668669 2022 tablet_replica.cc:439] T 892f9fad56544b6899146ea3ca42f0f2 P 50155961724d43b585d6e12c12bab09a: TxnStatusTablet state changed. Reason: New leader 7bda7656bed9450db86559665a6cc3fd. Latest consensus state: current_term: 1 leader_uuid: "7bda7656bed9450db86559665a6cc3fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50155961724d43b585d6e12c12bab09a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44123 } } peers { permanent_uuid: "7bda7656bed9450db86559665a6cc3fd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34007 } } peers { permanent_uuid: "e2fbec6f2e574d4bbfcf0fdf050449c5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35531 } } }
I20260924 09:55:50.668972 2022 tablet_replica.cc:442] T 892f9fad56544b6899146ea3ca42f0f2 P 50155961724d43b585d6e12c12bab09a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 09:55:50.675689 2034 tablet_replica.cc:439] T 892f9fad56544b6899146ea3ca42f0f2 P 7bda7656bed9450db86559665a6cc3fd: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7bda7656bed9450db86559665a6cc3fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50155961724d43b585d6e12c12bab09a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44123 } } peers { permanent_uuid: "7bda7656bed9450db86559665a6cc3fd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34007 } } peers { permanent_uuid: "e2fbec6f2e574d4bbfcf0fdf050449c5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35531 } } }
I20260924 09:55:50.675987 2034 tablet_replica.cc:442] T 892f9fad56544b6899146ea3ca42f0f2 P 7bda7656bed9450db86559665a6cc3fd: This TxnStatusTablet replica's current role is: LEADER
I20260924 09:55:50.676723 2023 tablet_replica.cc:439] T 892f9fad56544b6899146ea3ca42f0f2 P e2fbec6f2e574d4bbfcf0fdf050449c5: TxnStatusTablet state changed. Reason: New leader 7bda7656bed9450db86559665a6cc3fd. Latest consensus state: current_term: 1 leader_uuid: "7bda7656bed9450db86559665a6cc3fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50155961724d43b585d6e12c12bab09a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44123 } } peers { permanent_uuid: "7bda7656bed9450db86559665a6cc3fd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34007 } } peers { permanent_uuid: "e2fbec6f2e574d4bbfcf0fdf050449c5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35531 } } }
I20260924 09:55:50.676995 2023 tablet_replica.cc:442] T 892f9fad56544b6899146ea3ca42f0f2 P e2fbec6f2e574d4bbfcf0fdf050449c5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 09:55:50.678778 2022 tablet_replica.cc:439] T 892f9fad56544b6899146ea3ca42f0f2 P 50155961724d43b585d6e12c12bab09a: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7bda7656bed9450db86559665a6cc3fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50155961724d43b585d6e12c12bab09a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44123 } } peers { permanent_uuid: "7bda7656bed9450db86559665a6cc3fd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34007 } } peers { permanent_uuid: "e2fbec6f2e574d4bbfcf0fdf050449c5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35531 } } }
I20260924 09:55:50.679003 2022 tablet_replica.cc:442] T 892f9fad56544b6899146ea3ca42f0f2 P 50155961724d43b585d6e12c12bab09a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 09:55:50.680398 2020 tablet_replica.cc:439] T 892f9fad56544b6899146ea3ca42f0f2 P 7bda7656bed9450db86559665a6cc3fd: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7bda7656bed9450db86559665a6cc3fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50155961724d43b585d6e12c12bab09a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44123 } } peers { permanent_uuid: "7bda7656bed9450db86559665a6cc3fd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34007 } } peers { permanent_uuid: "e2fbec6f2e574d4bbfcf0fdf050449c5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35531 } } }
I20260924 09:55:50.680688 2020 tablet_replica.cc:442] T 892f9fad56544b6899146ea3ca42f0f2 P 7bda7656bed9450db86559665a6cc3fd: This TxnStatusTablet replica's current role is: LEADER
I20260924 09:55:50.715457 2023 tablet_replica.cc:439] T 892f9fad56544b6899146ea3ca42f0f2 P e2fbec6f2e574d4bbfcf0fdf050449c5: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7bda7656bed9450db86559665a6cc3fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50155961724d43b585d6e12c12bab09a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44123 } } peers { permanent_uuid: "7bda7656bed9450db86559665a6cc3fd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34007 } } peers { permanent_uuid: "e2fbec6f2e574d4bbfcf0fdf050449c5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35531 } } }
I20260924 09:55:50.715770 2023 tablet_replica.cc:442] T 892f9fad56544b6899146ea3ca42f0f2 P e2fbec6f2e574d4bbfcf0fdf050449c5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 09:55:50.723043 2073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:55:50.699248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39228 (local address 127.0.97.3:35531)
0924 09:55:50.711926 (+ 12678us) server_negotiation.cc:206] Beginning negotiation
0924 09:55:50.711933 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 09:55:50.712954 (+ 1021us) server_negotiation.cc:407] Connection header received
0924 09:55:50.713101 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:55:50.713108 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:55:50.713229 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:55:50.713413 (+ 184us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 09:55:50.714291 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:55:50.716510 (+ 2219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:55:50.720403 (+ 3893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:55:50.721051 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:55:50.721183 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:55:50.721573 (+ 390us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 09:55:50.722354 (+ 781us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 09:55:50.722509 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0924 09:55:50.722596 (+ 87us) server_negotiation.cc:299] Negotiation successful
0924 09:55:50.722691 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12520,"thread_start_us":13582,"threads_started":1}
I20260924 09:55:50.738349 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:55:50.732862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:44531 (local address 127.0.97.2:34007)
0924 09:55:50.733028 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0924 09:55:50.733032 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 09:55:50.733508 (+ 476us) server_negotiation.cc:407] Connection header received
0924 09:55:50.733674 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:55:50.733677 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:55:50.733748 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:55:50.733824 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 09:55:50.734691 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:55:50.735479 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:55:50.736769 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:55:50.737197 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:55:50.737307 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:55:50.737677 (+ 370us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 09:55:50.737806 (+ 129us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 09:55:50.737882 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 09:55:50.737996 (+ 114us) server_negotiation.cc:299] Negotiation successful
0924 09:55:50.738074 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 09:55:50.738435 2077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:55:50.732883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:34007 (local address 127.0.97.3:44531)
0924 09:55:50.733408 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0924 09:55:50.733425 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 09:55:50.733555 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:55:50.733843 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:55:50.733847 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:55:50.733859 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 09:55:50.734545 (+ 686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:55:50.734555 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:55:50.735590 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:55:50.735596 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:55:50.736617 (+ 1021us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:55:50.736629 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:55:50.737356 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:55:50.737363 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:55:50.737474 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:55:50.737525 (+ 51us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 09:55:50.737891 (+ 366us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 09:55:50.737902 (+ 11us) client_negotiation.cc:769] Sending connection context
0924 09:55:50.737968 (+ 66us) client_negotiation.cc:240] Negotiation successful
0924 09:55:50.738092 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":255,"threads_started":1}
I20260924 09:55:50.751307 1672 raft_consensus.cc:1261] T 33524144dfbd4a50bdb4cb485de943ad P 50155961724d43b585d6e12c12bab09a [term 1 FOLLOWER]: Refusing update from remote peer e2fbec6f2e574d4bbfcf0fdf050449c5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 09:55:50.751648 1812 raft_consensus.cc:1261] T 33524144dfbd4a50bdb4cb485de943ad P 7bda7656bed9450db86559665a6cc3fd [term 1 FOLLOWER]: Refusing update from remote peer e2fbec6f2e574d4bbfcf0fdf050449c5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 09:55:50.751987 2047 consensus_queue.cc:1035] T 33524144dfbd4a50bdb4cb485de943ad P e2fbec6f2e574d4bbfcf0fdf050449c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "50155961724d43b585d6e12c12bab09a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 09:55:50.752344 2023 consensus_queue.cc:1035] T 33524144dfbd4a50bdb4cb485de943ad P e2fbec6f2e574d4bbfcf0fdf050449c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7bda7656bed9450db86559665a6cc3fd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 09:55:50.756345 2063 mvcc.cc:205] Tried to move back new op lower bound from 7332838403071438848 to 7332838401752563712. Current Snapshot: MvccSnapshot[applied={T|T < 7332838403071438848}]
I20260924 09:55:50.757064 2064 mvcc.cc:205] Tried to move back new op lower bound from 7332838403071438848 to 7332838401752563712. Current Snapshot: MvccSnapshot[applied={T|T < 7332838403071438848}]
I20260924 09:55:50.761221 2062 mvcc.cc:205] Tried to move back new op lower bound from 7332838403071438848 to 7332838401752563712. Current Snapshot: MvccSnapshot[applied={T|T < 7332838403071438848}]
I20260924 09:55:50.786418 2073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:55:50.780223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:59535 (local address 127.0.97.3:35531)
0924 09:55:50.780378 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0924 09:55:50.780384 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 09:55:50.780712 (+ 328us) server_negotiation.cc:407] Connection header received
0924 09:55:50.780945 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:55:50.780950 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:55:50.781028 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:55:50.781114 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 09:55:50.782121 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:55:50.783225 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:55:50.784750 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:55:50.785310 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:55:50.785412 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:55:50.785769 (+ 357us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 09:55:50.785863 (+ 94us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 09:55:50.785931 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 09:55:50.786049 (+ 118us) server_negotiation.cc:299] Negotiation successful
0924 09:55:50.786120 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 09:55:50.786585 2082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:55:50.779989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:35531 (local address 127.0.97.2:59535)
0924 09:55:50.780608 (+ 619us) negotiation.cc:107] Waiting for socket to connect
0924 09:55:50.780626 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 09:55:50.780771 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:55:50.781128 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:55:50.781133 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:55:50.781145 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 09:55:50.781976 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:55:50.781988 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:55:50.783369 (+ 1381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:55:50.783373 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:55:50.784632 (+ 1259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:55:50.784644 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:55:50.785447 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:55:50.785453 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:55:50.785570 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:55:50.785619 (+ 49us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 09:55:50.785947 (+ 328us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 09:55:50.785958 (+ 11us) client_negotiation.cc:769] Sending connection context
0924 09:55:50.786022 (+ 64us) client_negotiation.cc:240] Negotiation successful
0924 09:55:50.786162 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":405,"threads_started":1}
I20260924 09:55:50.824199 1512 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57838:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 09:55:50.824477 1512 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57838:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 09:55:50.827903 1512 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9e23454a67c54530a74ec880bbe721d4: Sending DeleteTablet for 3 replicas of tablet 33524144dfbd4a50bdb4cb485de943ad
I20260924 09:55:50.828696 1647 tablet_service.cc:1514] Processing DeleteTablet for tablet 33524144dfbd4a50bdb4cb485de943ad with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 09:55:50 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:57902
I20260924 09:55:50.828955 1787 tablet_service.cc:1514] Processing DeleteTablet for tablet 33524144dfbd4a50bdb4cb485de943ad with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 09:55:50 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:59360
I20260924 09:55:50.829592 1926 tablet_service.cc:1514] Processing DeleteTablet for tablet 33524144dfbd4a50bdb4cb485de943ad with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 09:55:50 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:39216
I20260924 09:55:50.838835 2086 tablet_replica.cc:330] T 33524144dfbd4a50bdb4cb485de943ad P 7bda7656bed9450db86559665a6cc3fd: stopping tablet replica
I20260924 09:55:50.839632 2086 raft_consensus.cc:2229] T 33524144dfbd4a50bdb4cb485de943ad P 7bda7656bed9450db86559665a6cc3fd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 09:55:50.840273 2086 raft_consensus.cc:2258] T 33524144dfbd4a50bdb4cb485de943ad P 7bda7656bed9450db86559665a6cc3fd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 09:55:50.842216 2086 ts_tablet_manager.cc:1905] T 33524144dfbd4a50bdb4cb485de943ad P 7bda7656bed9450db86559665a6cc3fd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 09:55:50.845845 2087 tablet_replica.cc:330] T 33524144dfbd4a50bdb4cb485de943ad P e2fbec6f2e574d4bbfcf0fdf050449c5: stopping tablet replica
I20260924 09:55:50.846046 2087 raft_consensus.cc:2229] T 33524144dfbd4a50bdb4cb485de943ad P e2fbec6f2e574d4bbfcf0fdf050449c5 [term 1 LEADER]: Raft consensus shutting down.
I20260924 09:55:50.846692 2087 raft_consensus.cc:2258] T 33524144dfbd4a50bdb4cb485de943ad P e2fbec6f2e574d4bbfcf0fdf050449c5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 09:55:50.848017 2087 ts_tablet_manager.cc:1905] T 33524144dfbd4a50bdb4cb485de943ad P e2fbec6f2e574d4bbfcf0fdf050449c5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 09:55:50.849356 2085 tablet_replica.cc:330] T 33524144dfbd4a50bdb4cb485de943ad P 50155961724d43b585d6e12c12bab09a: stopping tablet replica
I20260924 09:55:50.849558 2085 raft_consensus.cc:2229] T 33524144dfbd4a50bdb4cb485de943ad P 50155961724d43b585d6e12c12bab09a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 09:55:50.851926 2085 raft_consensus.cc:2258] T 33524144dfbd4a50bdb4cb485de943ad P 50155961724d43b585d6e12c12bab09a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 09:55:50.853650 2085 ts_tablet_manager.cc:1905] T 33524144dfbd4a50bdb4cb485de943ad P 50155961724d43b585d6e12c12bab09a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 09:55:50.854103 2086 ts_tablet_manager.cc:1918] T 33524144dfbd4a50bdb4cb485de943ad P 7bda7656bed9450db86559665a6cc3fd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 09:55:50.854301 2086 log.cc:1196] T 33524144dfbd4a50bdb4cb485de943ad P 7bda7656bed9450db86559665a6cc3fd: Deleting WAL directory at /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790243735634835-388-0/minicluster-data/ts-1/wal/wals/33524144dfbd4a50bdb4cb485de943ad
I20260924 09:55:50.854821 2086 ts_tablet_manager.cc:1939] T 33524144dfbd4a50bdb4cb485de943ad P 7bda7656bed9450db86559665a6cc3fd: Deleting consensus metadata
I20260924 09:55:50.856026 1496 catalog_manager.cc:4841] TS 7bda7656bed9450db86559665a6cc3fd (127.0.97.2:34007): tablet 33524144dfbd4a50bdb4cb485de943ad (table test-table [id=93782efcaa0943eb801a23c7c454b312]) successfully deleted
I20260924 09:55:50.860337 2085 ts_tablet_manager.cc:1918] T 33524144dfbd4a50bdb4cb485de943ad P 50155961724d43b585d6e12c12bab09a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 09:55:50.860466 2085 log.cc:1196] T 33524144dfbd4a50bdb4cb485de943ad P 50155961724d43b585d6e12c12bab09a: Deleting WAL directory at /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790243735634835-388-0/minicluster-data/ts-0/wal/wals/33524144dfbd4a50bdb4cb485de943ad
I20260924 09:55:50.860879 2085 ts_tablet_manager.cc:1939] T 33524144dfbd4a50bdb4cb485de943ad P 50155961724d43b585d6e12c12bab09a: Deleting consensus metadata
I20260924 09:55:50.861901 1499 catalog_manager.cc:4841] TS 50155961724d43b585d6e12c12bab09a (127.0.97.1:44123): tablet 33524144dfbd4a50bdb4cb485de943ad (table test-table [id=93782efcaa0943eb801a23c7c454b312]) successfully deleted
I20260924 09:55:50.866377 2087 ts_tablet_manager.cc:1918] T 33524144dfbd4a50bdb4cb485de943ad P e2fbec6f2e574d4bbfcf0fdf050449c5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 09:55:50.866519 2087 log.cc:1196] T 33524144dfbd4a50bdb4cb485de943ad P e2fbec6f2e574d4bbfcf0fdf050449c5: Deleting WAL directory at /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790243735634835-388-0/minicluster-data/ts-2/wal/wals/33524144dfbd4a50bdb4cb485de943ad
I20260924 09:55:50.866940 2087 ts_tablet_manager.cc:1939] T 33524144dfbd4a50bdb4cb485de943ad P e2fbec6f2e574d4bbfcf0fdf050449c5: Deleting consensus metadata
I20260924 09:55:50.868042 1497 catalog_manager.cc:4841] TS e2fbec6f2e574d4bbfcf0fdf050449c5 (127.0.97.3:35531): tablet 33524144dfbd4a50bdb4cb485de943ad (table test-table [id=93782efcaa0943eb801a23c7c454b312]) successfully deleted
Sep 24 09:55:50 dist-test-slave-w8d2 krb5kdc[1467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243749, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260924 09:55:50.878654 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:55:50.840447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57916 (local address 127.0.97.1:44123)
0924 09:55:50.840639 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0924 09:55:50.840645 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 09:55:50.851848 (+ 11203us) server_negotiation.cc:407] Connection header received
0924 09:55:50.859683 (+ 7835us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:55:50.859689 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:55:50.859772 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:55:50.859845 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:55:50.862210 (+ 2365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:55:50.863318 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:55:50.865167 (+ 1849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:55:50.865797 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:55:50.865894 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:55:50.871501 (+ 5607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:55:50.871518 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:55:50.871538 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:55:50.871580 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:55:50.875301 (+ 3721us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:55:50.876448 (+ 1147us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:55:50.876459 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:55:50.876464 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:55:50.876625 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:55:50.877000 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:55:50.877005 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:55:50.877009 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:55:50.877458 (+ 449us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:55:50.877756 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0924 09:55:50.878024 (+ 268us) server_negotiation.cc:299] Negotiation successful
0924 09:55:50.878259 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
W20260924 09:55:50.879424 1713 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57916
I20260924 09:55:50.913429 2055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:55:50.896687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57866 (local address 127.0.97.62:42407)
0924 09:55:50.896919 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0924 09:55:50.896926 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 09:55:50.897399 (+ 473us) server_negotiation.cc:407] Connection header received
0924 09:55:50.897703 (+ 304us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:55:50.897708 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:55:50.897782 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:55:50.897854 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:55:50.899869 (+ 2015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:55:50.900936 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:55:50.902702 (+ 1766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:55:50.903399 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:55:50.903492 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:55:50.906922 (+ 3430us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:55:50.906929 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:55:50.906933 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:55:50.906972 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:55:50.910271 (+ 3299us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:55:50.911086 (+ 815us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:55:50.911091 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:55:50.911095 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:55:50.911250 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:55:50.911694 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:55:50.911699 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:55:50.911703 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:55:50.912042 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:55:50.912245 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0924 09:55:50.912861 (+ 616us) server_negotiation.cc:299] Negotiation successful
0924 09:55:50.913117 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
W20260924 09:55:50.915053 1512 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57866
I20260924 09:55:50.917505 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu pid 1583
I20260924 09:55:50.945591 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:55:50.918913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57932 (local address 127.0.97.1:44123)
0924 09:55:50.919079 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0924 09:55:50.919085 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 09:55:50.929172 (+ 10087us) server_negotiation.cc:407] Connection header received
0924 09:55:50.929628 (+ 456us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:55:50.929635 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:55:50.929731 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:55:50.929800 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:55:50.931694 (+ 1894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:55:50.932891 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:55:50.934679 (+ 1788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:55:50.935367 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:55:50.935458 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:55:50.938394 (+ 2936us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:55:50.938400 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:55:50.938405 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:55:50.938443 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:55:50.941986 (+ 3543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:55:50.942857 (+ 871us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:55:50.942863 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:55:50.942867 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:55:50.943058 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:55:50.943510 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:55:50.943516 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:55:50.943519 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:55:50.943843 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:55:50.944076 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0924 09:55:50.944599 (+ 523us) server_negotiation.cc:299] Negotiation successful
0924 09:55:50.944820 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
W20260924 09:55:50.947188 1713 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57932
W20260924 09:55:50.947866 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu pid 1583: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 09:55:50.948073 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu with pid 1583
I20260924 09:55:50.969630 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu pid 1722
Sep 24 09:55:50 dist-test-slave-w8d2 krb5kdc[1467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243749, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260924 09:55:50.988906 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:55:50.970719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59382 (local address 127.0.97.2:34007)
0924 09:55:50.970908 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0924 09:55:50.970913 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 09:55:50.970933 (+ 20us) server_negotiation.cc:407] Connection header received
0924 09:55:50.971050 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:55:50.971055 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:55:50.971134 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:55:50.971390 (+ 256us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:55:50.972861 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:55:50.974015 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:55:50.975890 (+ 1875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:55:50.976551 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:55:50.976709 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:55:50.981759 (+ 5050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:55:50.981780 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:55:50.981804 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:55:50.981848 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:55:50.985781 (+ 3933us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:55:50.986592 (+ 811us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:55:50.986601 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:55:50.986605 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:55:50.986795 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:55:50.987221 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:55:50.987226 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:55:50.987249 (+ 23us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:55:50.987686 (+ 437us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:55:50.987988 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0924 09:55:50.988301 (+ 313us) server_negotiation.cc:299] Negotiation successful
0924 09:55:50.988538 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
W20260924 09:55:50.989645 1852 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59382
W20260924 09:55:50.990388 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu pid 1722: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 09:55:50.990541 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu with pid 1722
I20260924 09:55:51.019112 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu pid 1861
Sep 24 09:55:51 dist-test-slave-w8d2 krb5kdc[1467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243749, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260924 09:55:51.035128 2073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:55:51.019862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39234 (local address 127.0.97.3:35531)
0924 09:55:51.020027 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0924 09:55:51.020031 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 09:55:51.020076 (+ 45us) server_negotiation.cc:407] Connection header received
0924 09:55:51.020257 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:55:51.020261 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:55:51.020342 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:55:51.020427 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:55:51.021877 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:55:51.022627 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:55:51.024427 (+ 1800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:55:51.024842 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:55:51.024934 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:55:51.029715 (+ 4781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:55:51.029729 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:55:51.029746 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:55:51.029777 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:55:51.032585 (+ 2808us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:55:51.033332 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:55:51.033338 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:55:51.033340 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:55:51.033457 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:55:51.033793 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:55:51.033796 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:55:51.033798 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:55:51.034122 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:55:51.034338 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0924 09:55:51.034666 (+ 328us) server_negotiation.cc:299] Negotiation successful
0924 09:55:51.034835 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
W20260924 09:55:51.035879 1991 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39234
W20260924 09:55:51.036446 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu pid 1861: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 09:55:51.036612 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu with pid 1861
I20260924 09:55:51.051429 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu pid 1480
I20260924 09:55:51.052459 1552 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 09:55:51.161880 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu with pid 1480
2026-09-24T09:55:51Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (4361 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790243735634835-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 09:55:51 dist-test-slave-w8d2 krb5kdc[2096](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 09:55:51 dist-test-slave-w8d2 krb5kdc[2096](info): set up 2 sockets
Sep 24 09:55:51 dist-test-slave-w8d2 krb5kdc[2096](info): commencing operation
krb5kdc: starting...
W20260924 09:55:52.135123 388 mini_kdc.cc:121] Time spent starting KDC: real 0.936s user 0.005s sys 0.009s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790243735634835-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790243735634835-388-0/krb5kdc/test-user.keytab.
Sep 24 09:55:52 dist-test-slave-w8d2 krb5kdc[2096](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243752, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T09:55:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T09:55:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790243735634835-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790243735634835-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260924 09:55:52.300509 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
/tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790243735634835-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790243735634835-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790243735634835-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790243735634835-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42045
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790243735634835-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:42045
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790243735634835-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790243735634835-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790243735634835-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 09:55:52.601608 2109 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 09:55:52.601889 2109 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 09:55:52.602015 2109 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 09:55:52.609126 2109 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 09:55:52.609226 2109 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260924 09:55:52.609289 2109 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 09:55:52.609380 2109 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 09:55:52.609440 2109 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 09:55:52.609521 2109 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 09:55:52.619089 2109 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790243735634835-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790243735634835-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:42045
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790243735634835-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42045
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790243735634835-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790243735634835-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 09:55:52.620337 2109 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 09:55:52.621627 2109 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 09:55:52.632349 2115 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:55:52.632321 2117 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:55:52.632586 2114 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 09:55:52.632972 2109 server_base.cc:1034] running on GCE node
I20260924 09:55:52.633740 2109 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 09:55:52.635061 2109 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 09:55:52.636255 2109 hybrid_clock.cc:648] HybridClock initialized: now 1790243752636184 us; error 50 us; skew 500 ppm
I20260924 09:55:52.636580 2109 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 09:55:52 dist-test-slave-w8d2 krb5kdc[2096](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243752, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 09:55:52.639580 2109 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260924 09:55:52.640903 2109 webserver.cc:460] Webserver started at http://127.0.97.62:42243/ using document root <none> and password file <none>
I20260924 09:55:52.641563 2109 fs_manager.cc:359] Metadata directory not provided
I20260924 09:55:52.641655 2109 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 09:55:52.641942 2109 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 09:55:52.644820 2109 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790243735634835-388-0/minicluster-data/master-0/data/instance:
uuid: "fcbcd495a6484513824d800e6faedb15"
format_stamp: "Formatted at 2026-09-24 09:55:52 on dist-test-slave-w8d2"
server_key: "b63428a6072b59442a2cdc11bffac028"
server_key_iv: "194fa4d972e80c737cef73c05027268a"
server_key_version: "encryptionkey@0"
I20260924 09:55:52.645521 2109 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790243735634835-388-0/minicluster-data/master-0/wal/instance:
uuid: "fcbcd495a6484513824d800e6faedb15"
format_stamp: "Formatted at 2026-09-24 09:55:52 on dist-test-slave-w8d2"
server_key: "b63428a6072b59442a2cdc11bffac028"
server_key_iv: "194fa4d972e80c737cef73c05027268a"
server_key_version: "encryptionkey@0"
I20260924 09:55:52.650149 2109 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260924 09:55:52.653225 2124 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 09:55:52.654476 2109 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 09:55:52.654618 2109 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790243735634835-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790243735634835-388-0/minicluster-data/master-0/wal
uuid: "fcbcd495a6484513824d800e6faedb15"
format_stamp: "Formatted at 2026-09-24 09:55:52 on dist-test-slave-w8d2"
server_key: "b63428a6072b59442a2cdc11bffac028"
server_key_iv: "194fa4d972e80c737cef73c05027268a"
server_key_version: "encryptionkey@0"
I20260924 09:55:52.654793 2109 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790243735634835-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790243735634835-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790243735634835-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 09:55:52.668010 2109 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 09:55:52.671795 2109 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 09:55:52.672021 2109 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 09:55:52.691164 2109 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:42045
I20260924 09:55:52.691182 2185 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:42045 every 8 connection(s)
I20260924 09:55:52.692481 2109 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790243735634835-388-0/minicluster-data/master-0/data/info.pb
I20260924 09:55:52.695041 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu as pid 2109
I20260924 09:55:52.695271 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790243735634835-388-0/minicluster-data/master-0/wal/instance
I20260924 09:55:52.696017 388 external_mini_cluster.cc:1428] Setting key 9c1e028c2d01736e0006f63b95d0ea02
I20260924 09:55:52.696509 2186 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 09:55:52.709756 2186 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fcbcd495a6484513824d800e6faedb15: Bootstrap starting.
Sep 24 09:55:52 dist-test-slave-w8d2 krb5kdc[2096](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243752, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 09:55:52.713337 2186 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fcbcd495a6484513824d800e6faedb15: Neither blocks nor log segments found. Creating new log.
I20260924 09:55:52.714637 2186 log.cc:824] T 00000000000000000000000000000000 P fcbcd495a6484513824d800e6faedb15: Log is configured to *not* fsync() on all Append() calls
I20260924 09:55:52.717221 2186 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fcbcd495a6484513824d800e6faedb15: No bootstrap required, opened a new log
I20260924 09:55:52.718344 2189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:55:52.697946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50320 (local address 127.0.97.62:42045)
0924 09:55:52.698667 (+ 721us) server_negotiation.cc:206] Beginning negotiation
0924 09:55:52.698679 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 09:55:52.698717 (+ 38us) server_negotiation.cc:407] Connection header received
0924 09:55:52.700810 (+ 2093us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:55:52.700832 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:55:52.701224 (+ 392us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:55:52.701555 (+ 331us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:55:52.703183 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:55:52.704441 (+ 1258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:55:52.706058 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:55:52.706754 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:55:52.706911 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:55:52.711072 (+ 4161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:55:52.711084 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:55:52.711102 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:55:52.711150 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:55:52.714669 (+ 3519us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:55:52.715302 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:55:52.715308 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:55:52.715318 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:55:52.715479 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:55:52.715810 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:55:52.715815 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:55:52.715817 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:55:52.716331 (+ 514us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:55:52.716576 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0924 09:55:52.716982 (+ 406us) server_negotiation.cc:299] Negotiation successful
0924 09:55:52.717384 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":507,"thread_start_us":337,"threads_started":1}
I20260924 09:55:52.724998 2186 raft_consensus.cc:348] T 00000000000000000000000000000000 P fcbcd495a6484513824d800e6faedb15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcbcd495a6484513824d800e6faedb15" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42045 } }
I20260924 09:55:52.725323 2186 raft_consensus.cc:374] T 00000000000000000000000000000000 P fcbcd495a6484513824d800e6faedb15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 09:55:52.725425 2186 raft_consensus.cc:729] T 00000000000000000000000000000000 P fcbcd495a6484513824d800e6faedb15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fcbcd495a6484513824d800e6faedb15, State: Initialized, Role: FOLLOWER
I20260924 09:55:52.726016 2186 consensus_queue.cc:260] T 00000000000000000000000000000000 P fcbcd495a6484513824d800e6faedb15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcbcd495a6484513824d800e6faedb15" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42045 } }
I20260924 09:55:52.726275 2186 raft_consensus.cc:388] T 00000000000000000000000000000000 P fcbcd495a6484513824d800e6faedb15 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 09:55:52.726420 2186 raft_consensus.cc:482] T 00000000000000000000000000000000 P fcbcd495a6484513824d800e6faedb15 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 09:55:52.726583 2186 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fcbcd495a6484513824d800e6faedb15 [term 0 FOLLOWER]: Advancing to term 1
I20260924 09:55:52.728595 2186 raft_consensus.cc:504] T 00000000000000000000000000000000 P fcbcd495a6484513824d800e6faedb15 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcbcd495a6484513824d800e6faedb15" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42045 } }
I20260924 09:55:52.729022 2186 leader_election.cc:302] T 00000000000000000000000000000000 P fcbcd495a6484513824d800e6faedb15 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fcbcd495a6484513824d800e6faedb15; no voters:
I20260924 09:55:52.729516 2186 leader_election.cc:288] T 00000000000000000000000000000000 P fcbcd495a6484513824d800e6faedb15 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 09:55:52.729626 2191 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fcbcd495a6484513824d800e6faedb15 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 09:55:52.730219 2191 raft_consensus.cc:686] T 00000000000000000000000000000000 P fcbcd495a6484513824d800e6faedb15 [term 1 LEADER]: Becoming Leader. State: Replica: fcbcd495a6484513824d800e6faedb15, State: Running, Role: LEADER
I20260924 09:55:52.730698 2191 consensus_queue.cc:237] T 00000000000000000000000000000000 P fcbcd495a6484513824d800e6faedb15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcbcd495a6484513824d800e6faedb15" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42045 } }
I20260924 09:55:52.731460 2186 sys_catalog.cc:564] T 00000000000000000000000000000000 P fcbcd495a6484513824d800e6faedb15 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 09:55:52.733389 2192 sys_catalog.cc:455] T 00000000000000000000000000000000 P fcbcd495a6484513824d800e6faedb15 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fcbcd495a6484513824d800e6faedb15" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcbcd495a6484513824d800e6faedb15" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42045 } } }
I20260924 09:55:52.733654 2192 sys_catalog.cc:458] T 00000000000000000000000000000000 P fcbcd495a6484513824d800e6faedb15 [sys.catalog]: This master's current role is: LEADER
I20260924 09:55:52.733894 2193 sys_catalog.cc:455] T 00000000000000000000000000000000 P fcbcd495a6484513824d800e6faedb15 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fcbcd495a6484513824d800e6faedb15. Latest consensus state: current_term: 1 leader_uuid: "fcbcd495a6484513824d800e6faedb15" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcbcd495a6484513824d800e6faedb15" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42045 } } }
I20260924 09:55:52.734148 2193 sys_catalog.cc:458] T 00000000000000000000000000000000 P fcbcd495a6484513824d800e6faedb15 [sys.catalog]: This master's current role is: LEADER
I20260924 09:55:52.735937 2197 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 09:55:52.740829 2197 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 09:55:52.746790 2208 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260924 09:55:52.751121 2197 catalog_manager.cc:1347] Generated new cluster ID: be765600f3944d8f8c006ca3a34a373b
I20260924 09:55:52.751214 2197 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 09:55:52.764544 2197 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 09:55:52.765817 2197 catalog_manager.cc:1504] Loading token signing keys...
I20260924 09:55:52.773200 2197 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fcbcd495a6484513824d800e6faedb15: Generated new TSK 0
I20260924 09:55:52.773973 2197 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790243735634835-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790243735634835-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260924 09:55:52.825009 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
/tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790243735634835-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790243735634835-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790243735634835-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790243735634835-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42045
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790243735634835-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790243735634835-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790243735634835-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790243735634835-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 09:55:53.133229 2212 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 09:55:53.133508 2212 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 09:55:53.133646 2212 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 09:55:53.140605 2212 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 09:55:53.140717 2212 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 09:55:53.140839 2212 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260924 09:55:53.140897 2212 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 09:55:53.140995 2212 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260924 09:55:53.150218 2212 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790243735634835-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790243735634835-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790243735634835-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790243735634835-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:42045
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790243735634835-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 09:55:53.151461 2212 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 09:55:53.152717 2212 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 09:55:53.164464 2218 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:55:53.164484 2220 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:55:53.164433 2217 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 09:55:53.165181 2212 server_base.cc:1034] running on GCE node
I20260924 09:55:53.165728 2212 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 09:55:53.166589 2212 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 09:55:53.167774 2212 hybrid_clock.cc:648] HybridClock initialized: now 1790243753167727 us; error 42 us; skew 500 ppm
I20260924 09:55:53.168068 2212 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 09:55:53 dist-test-slave-w8d2 krb5kdc[2096](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243753, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 09:55:53.173281 2212 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260924 09:55:53.174561 2212 webserver.cc:460] Webserver started at http://127.0.97.1:38047/ using document root <none> and password file <none>
I20260924 09:55:53.175225 2212 fs_manager.cc:359] Metadata directory not provided
I20260924 09:55:53.175380 2212 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 09:55:53.175654 2212 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 09:55:53.178511 2212 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790243735634835-388-0/minicluster-data/ts-0/data/instance:
uuid: "9ad27677628b497bb0f4e8636501851d"
format_stamp: "Formatted at 2026-09-24 09:55:53 on dist-test-slave-w8d2"
server_key: "db67926e0bfd604d923ddc8119dbd6ae"
server_key_iv: "99546d81f255f56ec52ec35ee3f5deb4"
server_key_version: "encryptionkey@0"
I20260924 09:55:53.179226 2212 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790243735634835-388-0/minicluster-data/ts-0/wal/instance:
uuid: "9ad27677628b497bb0f4e8636501851d"
format_stamp: "Formatted at 2026-09-24 09:55:53 on dist-test-slave-w8d2"
server_key: "db67926e0bfd604d923ddc8119dbd6ae"
server_key_iv: "99546d81f255f56ec52ec35ee3f5deb4"
server_key_version: "encryptionkey@0"
I20260924 09:55:53.183925 2212 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260924 09:55:53.186964 2227 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 09:55:53.188169 2212 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260924 09:55:53.188308 2212 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790243735634835-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790243735634835-388-0/minicluster-data/ts-0/wal
uuid: "9ad27677628b497bb0f4e8636501851d"
format_stamp: "Formatted at 2026-09-24 09:55:53 on dist-test-slave-w8d2"
server_key: "db67926e0bfd604d923ddc8119dbd6ae"
server_key_iv: "99546d81f255f56ec52ec35ee3f5deb4"
server_key_version: "encryptionkey@0"
I20260924 09:55:53.188498 2212 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790243735634835-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790243735634835-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790243735634835-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 09:55:53.206459 2212 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 09:55:53.210053 2212 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 09:55:53.210296 2212 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 09:55:53.212773 2212 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 09:55:53.212869 2212 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 09:55:53.212988 2212 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 09:55:53.213066 2212 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 09:55:53 dist-test-slave-w8d2 krb5kdc[2096](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243753, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 09:55:53.235137 2189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:55:53.213155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:43305 (local address 127.0.97.62:42045)
0924 09:55:53.213347 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0924 09:55:53.213355 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 09:55:53.214745 (+ 1390us) server_negotiation.cc:407] Connection header received
0924 09:55:53.217484 (+ 2739us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:55:53.217491 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:55:53.217581 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:55:53.217652 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:55:53.220261 (+ 2609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:55:53.221104 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:55:53.223910 (+ 2806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:55:53.224315 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:55:53.224415 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:55:53.228897 (+ 4482us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:55:53.228902 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:55:53.228905 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:55:53.228936 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:55:53.231783 (+ 2847us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:55:53.232694 (+ 911us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:55:53.232699 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:55:53.232702 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:55:53.232864 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:55:53.233381 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:55:53.233385 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:55:53.233387 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:55:53.233656 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:55:53.233797 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0924 09:55:53.234752 (+ 955us) server_negotiation.cc:299] Negotiation successful
0924 09:55:53.234913 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 09:55:53.236958 2236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:55:53.213743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42045 (local address 127.0.97.1:43305)
0924 09:55:53.214565 (+ 822us) negotiation.cc:107] Waiting for socket to connect
0924 09:55:53.214622 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0924 09:55:53.217137 (+ 2515us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:55:53.218231 (+ 1094us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:55:53.218245 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:55:53.219125 (+ 880us) client_negotiation.cc:189] Negotiated authn=SASL
0924 09:55:53.220069 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:55:53.220096 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:55:53.221275 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:55:53.221284 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:55:53.223741 (+ 2457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:55:53.223758 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:55:53.224554 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:55:53.224562 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:55:53.224871 (+ 309us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:55:53.225647 (+ 776us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 09:55:53.225679 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0924 09:55:53.228609 (+ 2930us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 09:55:53.231950 (+ 3341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 09:55:53.231958 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 09:55:53.231974 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 09:55:53.232575 (+ 601us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 09:55:53.233001 (+ 426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 09:55:53.233007 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 09:55:53.233011 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 09:55:53.233273 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 09:55:53.233778 (+ 505us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 09:55:53.233788 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 09:55:53.234317 (+ 529us) client_negotiation.cc:769] Sending connection context
0924 09:55:53.234669 (+ 352us) client_negotiation.cc:240] Negotiation successful
0924 09:55:53.235273 (+ 604us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":530,"thread_start_us":318,"threads_started":1}
I20260924 09:55:53.256152 2318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:55:53.242717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42045 (local address 127.0.97.1:43947)
0924 09:55:53.243309 (+ 592us) negotiation.cc:107] Waiting for socket to connect
0924 09:55:53.243334 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0924 09:55:53.243486 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:55:53.243826 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:55:53.243831 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:55:53.244243 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0924 09:55:53.244494 (+ 251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:55:53.244502 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:55:53.245396 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:55:53.245401 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:55:53.246360 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:55:53.246369 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:55:53.247058 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:55:53.247064 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:55:53.247207 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:55:53.247916 (+ 709us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 09:55:53.247943 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0924 09:55:53.248976 (+ 1033us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 09:55:53.252953 (+ 3977us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 09:55:53.252957 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 09:55:53.252961 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 09:55:53.253577 (+ 616us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 09:55:53.254021 (+ 444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 09:55:53.254026 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 09:55:53.254030 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 09:55:53.254178 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 09:55:53.254837 (+ 659us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 09:55:53.254843 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 09:55:53.255040 (+ 197us) client_negotiation.cc:769] Sending connection context
0924 09:55:53.255201 (+ 161us) client_negotiation.cc:240] Negotiation successful
0924 09:55:53.255562 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":306,"threads_started":1}
I20260924 09:55:53.256669 2189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:55:53.242683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:43947 (local address 127.0.97.62:42045)
0924 09:55:53.242819 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0924 09:55:53.242823 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 09:55:53.243425 (+ 602us) server_negotiation.cc:407] Connection header received
0924 09:55:53.243643 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:55:53.243649 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:55:53.243714 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:55:53.243791 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:55:53.244616 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:55:53.245281 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:55:53.246485 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:55:53.246897 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:55:53.246998 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:55:53.249163 (+ 2165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:55:53.249170 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:55:53.249175 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:55:53.249210 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:55:53.252711 (+ 3501us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:55:53.253725 (+ 1014us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:55:53.253730 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:55:53.253733 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:55:53.253892 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:55:53.254293 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:55:53.254299 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:55:53.254302 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:55:53.254629 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:55:53.254767 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0924 09:55:53.256105 (+ 1338us) server_negotiation.cc:299] Negotiation successful
0924 09:55:53.256324 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 09:55:53.267069 2346 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:44953 every 8 connection(s)
I20260924 09:55:53.267077 2212 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:44953
I20260924 09:55:53.268750 2212 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790243735634835-388-0/minicluster-data/ts-0/data/info.pb
I20260924 09:55:53.270749 2347 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42045
I20260924 09:55:53.271011 2347 heartbeater.cc:464] Registering TS with master...
I20260924 09:55:53.272032 2347 heartbeater.cc:511] Master 127.0.97.62:42045 requested a full tablet report, sending...
I20260924 09:55:53.273612 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu as pid 2212
I20260924 09:55:53.273825 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790243735634835-388-0/minicluster-data/ts-0/wal/instance
I20260924 09:55:53.274289 2141 ts_manager.cc:194] Registered new tserver with Master: 9ad27677628b497bb0f4e8636501851d (127.0.97.1:44953)
I20260924 09:55:53.274559 388 external_mini_cluster.cc:1428] Setting key f14db84421d74a67b817f6ab33f1fc84
I20260924 09:55:53.276229 2141 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:43305
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790243735634835-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790243735634835-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260924 09:55:53.315991 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
/tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790243735634835-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790243735634835-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790243735634835-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790243735634835-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42045
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790243735634835-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790243735634835-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790243735634835-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790243735634835-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 09:55:53.613185 2351 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 09:55:53.613457 2351 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 09:55:53.613590 2351 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 09:55:53.620431 2351 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 09:55:53.620522 2351 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 09:55:53.620637 2351 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260924 09:55:53.620694 2351 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 09:55:53.620792 2351 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260924 09:55:53.630167 2351 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790243735634835-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790243735634835-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790243735634835-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790243735634835-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:42045
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790243735634835-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 09:55:53.631471 2351 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 09:55:53.632743 2351 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 09:55:53.644515 2357 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:55:53.644582 2356 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:55:53.644555 2359 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 09:55:53.645300 2351 server_base.cc:1034] running on GCE node
I20260924 09:55:53.645861 2351 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 09:55:53.646694 2351 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 09:55:53.647877 2351 hybrid_clock.cc:648] HybridClock initialized: now 1790243753647830 us; error 36 us; skew 500 ppm
I20260924 09:55:53.648169 2351 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 09:55:53 dist-test-slave-w8d2 krb5kdc[2096](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243753, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 09:55:53.651002 2351 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260924 09:55:53.652341 2351 webserver.cc:460] Webserver started at http://127.0.97.2:35049/ using document root <none> and password file <none>
I20260924 09:55:53.652982 2351 fs_manager.cc:359] Metadata directory not provided
I20260924 09:55:53.653102 2351 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 09:55:53.653375 2351 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 09:55:53.656244 2351 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790243735634835-388-0/minicluster-data/ts-1/data/instance:
uuid: "ccac0f0c5aa54f1abe1d585545d2c34a"
format_stamp: "Formatted at 2026-09-24 09:55:53 on dist-test-slave-w8d2"
server_key: "051c5a8478be0eca9b09b671c63e7b6c"
server_key_iv: "bd5b1a6401ec9e1618c94bdd03db0121"
server_key_version: "encryptionkey@0"
I20260924 09:55:53.657015 2351 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790243735634835-388-0/minicluster-data/ts-1/wal/instance:
uuid: "ccac0f0c5aa54f1abe1d585545d2c34a"
format_stamp: "Formatted at 2026-09-24 09:55:53 on dist-test-slave-w8d2"
server_key: "051c5a8478be0eca9b09b671c63e7b6c"
server_key_iv: "bd5b1a6401ec9e1618c94bdd03db0121"
server_key_version: "encryptionkey@0"
I20260924 09:55:53.662024 2351 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 09:55:53.665269 2366 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 09:55:53.666486 2351 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 09:55:53.666625 2351 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790243735634835-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790243735634835-388-0/minicluster-data/ts-1/wal
uuid: "ccac0f0c5aa54f1abe1d585545d2c34a"
format_stamp: "Formatted at 2026-09-24 09:55:53 on dist-test-slave-w8d2"
server_key: "051c5a8478be0eca9b09b671c63e7b6c"
server_key_iv: "bd5b1a6401ec9e1618c94bdd03db0121"
server_key_version: "encryptionkey@0"
I20260924 09:55:53.666790 2351 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790243735634835-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790243735634835-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790243735634835-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 09:55:53.695540 2351 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 09:55:53.700449 2351 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 09:55:53.700727 2351 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 09:55:53.703729 2351 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 09:55:53.703847 2351 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 09:55:53.703941 2351 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 09:55:53.704011 2351 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 09:55:53 dist-test-slave-w8d2 krb5kdc[2096](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243753, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 09:55:53.725577 2189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:55:53.704104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:45875 (local address 127.0.97.62:42045)
0924 09:55:53.704328 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0924 09:55:53.704334 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 09:55:53.705755 (+ 1421us) server_negotiation.cc:407] Connection header received
0924 09:55:53.707851 (+ 2096us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:55:53.707857 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:55:53.707940 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:55:53.708011 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:55:53.710338 (+ 2327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:55:53.711500 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:55:53.714048 (+ 2548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:55:53.714601 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:55:53.714715 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:55:53.719345 (+ 4630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:55:53.719350 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:55:53.719354 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:55:53.719386 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:55:53.722354 (+ 2968us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:55:53.723328 (+ 974us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:55:53.723333 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:55:53.723335 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:55:53.723455 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:55:53.723912 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:55:53.723916 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:55:53.723919 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:55:53.724163 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:55:53.724271 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0924 09:55:53.725183 (+ 912us) server_negotiation.cc:299] Negotiation successful
0924 09:55:53.725348 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106}
I20260924 09:55:53.727425 2375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:55:53.704690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42045 (local address 127.0.97.2:45875)
0924 09:55:53.705585 (+ 895us) negotiation.cc:107] Waiting for socket to connect
0924 09:55:53.705636 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0924 09:55:53.707543 (+ 1907us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:55:53.708483 (+ 940us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:55:53.708494 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:55:53.709201 (+ 707us) client_negotiation.cc:189] Negotiated authn=SASL
0924 09:55:53.710117 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:55:53.710143 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:55:53.711725 (+ 1582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:55:53.711730 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:55:53.713871 (+ 2141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:55:53.713910 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:55:53.714834 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:55:53.714842 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:55:53.715115 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:55:53.716049 (+ 934us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 09:55:53.716083 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0924 09:55:53.718993 (+ 2910us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 09:55:53.722520 (+ 3527us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 09:55:53.722531 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 09:55:53.722552 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0924 09:55:53.723176 (+ 624us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 09:55:53.723549 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 09:55:53.723554 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 09:55:53.723559 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 09:55:53.723818 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 09:55:53.724273 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 09:55:53.724282 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 09:55:53.724768 (+ 486us) client_negotiation.cc:769] Sending connection context
0924 09:55:53.725067 (+ 299us) client_negotiation.cc:240] Negotiation successful
0924 09:55:53.725642 (+ 575us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":584,"thread_start_us":363,"threads_started":1}
I20260924 09:55:53.745852 2189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:55:53.732825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:43201 (local address 127.0.97.62:42045)
0924 09:55:53.732974 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0924 09:55:53.732977 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 09:55:53.733565 (+ 588us) server_negotiation.cc:407] Connection header received
0924 09:55:53.733695 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:55:53.733701 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:55:53.733785 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:55:53.733890 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:55:53.734695 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:55:53.735808 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:55:53.737594 (+ 1786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:55:53.738026 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:55:53.738147 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:55:53.740634 (+ 2487us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:55:53.740640 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:55:53.740645 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:55:53.740681 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:55:53.743056 (+ 2375us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:55:53.743893 (+ 837us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:55:53.743898 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:55:53.743900 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:55:53.744024 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:55:53.744398 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:55:53.744403 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:55:53.744406 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:55:53.744777 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:55:53.744903 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0924 09:55:53.745298 (+ 395us) server_negotiation.cc:299] Negotiation successful
0924 09:55:53.745508 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 09:55:53.745980 2441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:55:53.732899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42045 (local address 127.0.97.2:43201)
0924 09:55:53.733472 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0924 09:55:53.733491 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 09:55:53.733605 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:55:53.733898 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:55:53.733901 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:55:53.734298 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0924 09:55:53.734580 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:55:53.734590 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:55:53.735934 (+ 1344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:55:53.735940 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:55:53.737444 (+ 1504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:55:53.737459 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:55:53.738212 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:55:53.738220 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:55:53.738416 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:55:53.739319 (+ 903us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 09:55:53.739350 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0924 09:55:53.740453 (+ 1103us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 09:55:53.743166 (+ 2713us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 09:55:53.743172 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 09:55:53.743176 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 09:55:53.743783 (+ 607us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 09:55:53.744119 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 09:55:53.744128 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 09:55:53.744131 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 09:55:53.744296 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 09:55:53.744891 (+ 595us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 09:55:53.744896 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 09:55:53.745074 (+ 178us) client_negotiation.cc:769] Sending connection context
0924 09:55:53.745207 (+ 133us) client_negotiation.cc:240] Negotiation successful
0924 09:55:53.745545 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":298,"threads_started":1}
I20260924 09:55:53.753000 2351 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:35579
I20260924 09:55:53.753033 2485 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:35579 every 8 connection(s)
I20260924 09:55:53.754130 2351 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790243735634835-388-0/minicluster-data/ts-1/data/info.pb
I20260924 09:55:53.755867 2486 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42045
I20260924 09:55:53.756081 2486 heartbeater.cc:464] Registering TS with master...
I20260924 09:55:53.756691 2486 heartbeater.cc:511] Master 127.0.97.62:42045 requested a full tablet report, sending...
I20260924 09:55:53.757902 2141 ts_manager.cc:194] Registered new tserver with Master: ccac0f0c5aa54f1abe1d585545d2c34a (127.0.97.2:35579)
I20260924 09:55:53.758780 2141 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:45875
I20260924 09:55:53.759984 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu as pid 2351
I20260924 09:55:53.760192 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790243735634835-388-0/minicluster-data/ts-1/wal/instance
I20260924 09:55:53.760700 388 external_mini_cluster.cc:1428] Setting key 2f3670ae529424e0b1239c5bec145146
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790243735634835-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790243735634835-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260924 09:55:53.795943 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
/tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790243735634835-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790243735634835-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790243735634835-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790243735634835-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42045
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790243735634835-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790243735634835-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790243735634835-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790243735634835-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 09:55:54.099915 2490 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 09:55:54.100211 2490 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 09:55:54.100333 2490 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 09:55:54.107606 2490 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 09:55:54.107734 2490 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 09:55:54.107851 2490 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260924 09:55:54.107918 2490 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 09:55:54.108019 2490 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260924 09:55:54.117540 2490 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790243735634835-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790243735634835-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790243735634835-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790243735634835-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:42045
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790243735634835-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 09:55:54.118777 2490 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 09:55:54.120107 2490 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 09:55:54.132738 2498 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:55:54.132769 2496 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:55:54.132800 2495 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 09:55:54.133471 2490 server_base.cc:1034] running on GCE node
I20260924 09:55:54.134079 2490 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 09:55:54.134979 2490 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 09:55:54.136178 2490 hybrid_clock.cc:648] HybridClock initialized: now 1790243754136105 us; error 66 us; skew 500 ppm
I20260924 09:55:54.136508 2490 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 09:55:54 dist-test-slave-w8d2 krb5kdc[2096](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243754, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 09:55:54.139478 2490 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260924 09:55:54.140849 2490 webserver.cc:460] Webserver started at http://127.0.97.3:33601/ using document root <none> and password file <none>
I20260924 09:55:54.141503 2490 fs_manager.cc:359] Metadata directory not provided
I20260924 09:55:54.141599 2490 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 09:55:54.141841 2490 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 09:55:54.144734 2490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790243735634835-388-0/minicluster-data/ts-2/data/instance:
uuid: "e4ce48a623fa4741bed2e12aa3b7a452"
format_stamp: "Formatted at 2026-09-24 09:55:54 on dist-test-slave-w8d2"
server_key: "ea1b581fded1cfd0b480ebca26404e21"
server_key_iv: "f8fd67332163d6644c2af562f1b37c02"
server_key_version: "encryptionkey@0"
I20260924 09:55:54.145445 2490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790243735634835-388-0/minicluster-data/ts-2/wal/instance:
uuid: "e4ce48a623fa4741bed2e12aa3b7a452"
format_stamp: "Formatted at 2026-09-24 09:55:54 on dist-test-slave-w8d2"
server_key: "ea1b581fded1cfd0b480ebca26404e21"
server_key_iv: "f8fd67332163d6644c2af562f1b37c02"
server_key_version: "encryptionkey@0"
I20260924 09:55:54.150373 2490 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260924 09:55:54.153716 2505 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 09:55:54.155025 2490 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 09:55:54.155187 2490 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790243735634835-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790243735634835-388-0/minicluster-data/ts-2/wal
uuid: "e4ce48a623fa4741bed2e12aa3b7a452"
format_stamp: "Formatted at 2026-09-24 09:55:54 on dist-test-slave-w8d2"
server_key: "ea1b581fded1cfd0b480ebca26404e21"
server_key_iv: "f8fd67332163d6644c2af562f1b37c02"
server_key_version: "encryptionkey@0"
I20260924 09:55:54.155418 2490 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790243735634835-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790243735634835-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790243735634835-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 09:55:54.180017 2490 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 09:55:54.184031 2490 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 09:55:54.184305 2490 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 09:55:54.186878 2490 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 09:55:54.187026 2490 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 09:55:54.187150 2490 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 09:55:54.187248 2490 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 09:55:54 dist-test-slave-w8d2 krb5kdc[2096](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243754, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 09:55:54.208675 2189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:55:54.187713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:34269 (local address 127.0.97.62:42045)
0924 09:55:54.187892 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0924 09:55:54.187897 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 09:55:54.189377 (+ 1480us) server_negotiation.cc:407] Connection header received
0924 09:55:54.191445 (+ 2068us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:55:54.191451 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:55:54.191537 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:55:54.191659 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:55:54.194332 (+ 2673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:55:54.195295 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:55:54.197583 (+ 2288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:55:54.198092 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:55:54.198222 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:55:54.203023 (+ 4801us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:55:54.203030 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:55:54.203034 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:55:54.203077 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:55:54.205638 (+ 2561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:55:54.206312 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:55:54.206318 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:55:54.206321 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:55:54.206443 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:55:54.206842 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:55:54.206847 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:55:54.206851 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:55:54.207106 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:55:54.207255 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0924 09:55:54.208099 (+ 844us) server_negotiation.cc:299] Negotiation successful
0924 09:55:54.208326 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260924 09:55:54.210083 2514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:55:54.188393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42045 (local address 127.0.97.3:34269)
0924 09:55:54.189221 (+ 828us) negotiation.cc:107] Waiting for socket to connect
0924 09:55:54.189260 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 09:55:54.191061 (+ 1801us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:55:54.192095 (+ 1034us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:55:54.192111 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:55:54.193050 (+ 939us) client_negotiation.cc:189] Negotiated authn=SASL
0924 09:55:54.194096 (+ 1046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:55:54.194137 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:55:54.195444 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:55:54.195448 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:55:54.197360 (+ 1912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:55:54.197371 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:55:54.198353 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:55:54.198360 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:55:54.198702 (+ 342us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:55:54.199567 (+ 865us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 09:55:54.199602 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0924 09:55:54.202768 (+ 3166us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 09:55:54.205769 (+ 3001us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 09:55:54.205778 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 09:55:54.205796 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0924 09:55:54.206201 (+ 405us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 09:55:54.206548 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 09:55:54.206553 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 09:55:54.206557 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 09:55:54.206740 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 09:55:54.207216 (+ 476us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 09:55:54.207225 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 09:55:54.207670 (+ 445us) client_negotiation.cc:769] Sending connection context
0924 09:55:54.208003 (+ 333us) client_negotiation.cc:240] Negotiation successful
0924 09:55:54.208592 (+ 589us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":576,"thread_start_us":341,"threads_started":1}
I20260924 09:55:54.226620 2490 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:42817
I20260924 09:55:54.226636 2624 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:42817 every 8 connection(s)
I20260924 09:55:54.227963 2490 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790243735634835-388-0/minicluster-data/ts-2/data/info.pb
I20260924 09:55:54.230094 2625 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42045
I20260924 09:55:54.230409 2625 heartbeater.cc:464] Registering TS with master...
I20260924 09:55:54.231074 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu as pid 2490
I20260924 09:55:54.231361 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790243735634835-388-0/minicluster-data/ts-2/wal/instance
I20260924 09:55:54.231315 2625 heartbeater.cc:511] Master 127.0.97.62:42045 requested a full tablet report, sending...
I20260924 09:55:54.231366 2189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:55:54.216749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:41353 (local address 127.0.97.62:42045)
0924 09:55:54.216942 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0924 09:55:54.216948 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 09:55:54.217713 (+ 765us) server_negotiation.cc:407] Connection header received
0924 09:55:54.217893 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:55:54.217897 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:55:54.217955 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:55:54.218045 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:55:54.219329 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:55:54.220239 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:55:54.222487 (+ 2248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:55:54.222925 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:55:54.223022 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:55:54.225467 (+ 2445us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:55:54.225473 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:55:54.225478 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:55:54.225556 (+ 78us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:55:54.228233 (+ 2677us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:55:54.229135 (+ 902us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:55:54.229141 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:55:54.229144 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:55:54.229326 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:55:54.229712 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:55:54.229717 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:55:54.229720 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:55:54.230094 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:55:54.230256 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0924 09:55:54.230757 (+ 501us) server_negotiation.cc:299] Negotiation successful
0924 09:55:54.230955 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 09:55:54.231441 2599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:55:54.216731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42045 (local address 127.0.97.3:41353)
0924 09:55:54.217584 (+ 853us) negotiation.cc:107] Waiting for socket to connect
0924 09:55:54.217612 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0924 09:55:54.217774 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:55:54.218089 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:55:54.218093 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:55:54.218727 (+ 634us) client_negotiation.cc:189] Negotiated authn=SASL
0924 09:55:54.219146 (+ 419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:55:54.219160 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:55:54.220380 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:55:54.220386 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:55:54.222317 (+ 1931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:55:54.222336 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:55:54.223104 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:55:54.223110 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:55:54.223324 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:55:54.223997 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 09:55:54.224020 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 09:55:54.225275 (+ 1255us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 09:55:54.228359 (+ 3084us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 09:55:54.228364 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 09:55:54.228369 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 09:55:54.228972 (+ 603us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 09:55:54.229445 (+ 473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 09:55:54.229449 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 09:55:54.229451 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 09:55:54.229609 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 09:55:54.230310 (+ 701us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 09:55:54.230313 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 09:55:54.230489 (+ 176us) client_negotiation.cc:769] Sending connection context
0924 09:55:54.230639 (+ 150us) client_negotiation.cc:240] Negotiation successful
0924 09:55:54.230984 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":707,"thread_start_us":564,"threads_started":1}
I20260924 09:55:54.231890 388 external_mini_cluster.cc:1428] Setting key c0317235f4fbe5fa9eaac1e00c6a640b
I20260924 09:55:54.232998 2141 ts_manager.cc:194] Registered new tserver with Master: e4ce48a623fa4741bed2e12aa3b7a452 (127.0.97.3:42817)
I20260924 09:55:54.233948 2141 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:34269
I20260924 09:55:54.235131 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 09:55:54 dist-test-slave-w8d2 krb5kdc[2096](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243754, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 09:55:54 dist-test-slave-w8d2 krb5kdc[2096](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243754, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 09:55:54.278316 2189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:55:54.265290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48854 (local address 127.0.97.62:42045)
0924 09:55:54.265464 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0924 09:55:54.265470 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 09:55:54.265665 (+ 195us) server_negotiation.cc:407] Connection header received
0924 09:55:54.265863 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:55:54.265869 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:55:54.265987 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:55:54.266087 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:55:54.267351 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:55:54.268410 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:55:54.269573 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:55:54.270238 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:55:54.270372 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:55:54.273944 (+ 3572us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:55:54.273949 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:55:54.273952 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:55:54.273987 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:55:54.276374 (+ 2387us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:55:54.276934 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:55:54.276937 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:55:54.276940 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:55:54.277050 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:55:54.277313 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:55:54.277315 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:55:54.277317 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:55:54.277585 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:55:54.277751 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0924 09:55:54.277952 (+ 201us) server_negotiation.cc:299] Negotiation successful
0924 09:55:54.278105 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 09:55:54.279419 2347 heartbeater.cc:503] Master 127.0.97.62:42045 was elected leader, sending a full tablet report...
I20260924 09:55:54.282058 2140 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48854:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 09:55:54.288923 2140 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 09:55:54.311944 2638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:55:54.301025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44953 (local address 127.0.0.1:39514)
0924 09:55:54.303859 (+ 2834us) negotiation.cc:107] Waiting for socket to connect
0924 09:55:54.303897 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 09:55:54.304148 (+ 251us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:55:54.305016 (+ 868us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:55:54.305024 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:55:54.305054 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 09:55:54.305546 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:55:54.305560 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:55:54.307314 (+ 1754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:55:54.307319 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:55:54.309385 (+ 2066us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:55:54.309398 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:55:54.311082 (+ 1684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:55:54.311088 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:55:54.311253 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:55:54.311286 (+ 33us) client_negotiation.cc:769] Sending connection context
0924 09:55:54.311366 (+ 80us) client_negotiation.cc:240] Negotiation successful
0924 09:55:54.311488 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2617,"spinlock_wait_cycles":45824,"thread_start_us":2392,"threads_started":1}
I20260924 09:55:54.312062 2639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:55:54.300989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44566 (local address 127.0.97.3:42817)
0924 09:55:54.301720 (+ 731us) server_negotiation.cc:206] Beginning negotiation
0924 09:55:54.301730 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 09:55:54.303996 (+ 2266us) server_negotiation.cc:407] Connection header received
0924 09:55:54.304385 (+ 389us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:55:54.304393 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:55:54.304606 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:55:54.304757 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 09:55:54.305837 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:55:54.306974 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:55:54.309171 (+ 2197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:55:54.310563 (+ 1392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:55:54.310696 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:55:54.310922 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0924 09:55:54.311596 (+ 674us) server_negotiation.cc:299] Negotiation successful
0924 09:55:54.311766 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":547,"thread_start_us":355,"threads_started":1}
I20260924 09:55:54.312548 2641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:55:54.301515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39514 (local address 127.0.97.1:44953)
0924 09:55:54.301945 (+ 430us) server_negotiation.cc:206] Beginning negotiation
0924 09:55:54.301953 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 09:55:54.303993 (+ 2040us) server_negotiation.cc:407] Connection header received
0924 09:55:54.304390 (+ 397us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:55:54.304398 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:55:54.304589 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:55:54.304745 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 09:55:54.305724 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:55:54.307171 (+ 1447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:55:54.309532 (+ 2361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:55:54.310854 (+ 1322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:55:54.310967 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:55:54.311147 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0924 09:55:54.311420 (+ 273us) server_negotiation.cc:299] Negotiation successful
0924 09:55:54.311580 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":204,"threads_started":1}
I20260924 09:55:54.312721 2642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:55:54.304478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34976 (local address 127.0.97.2:35579)
0924 09:55:54.304996 (+ 518us) server_negotiation.cc:206] Beginning negotiation
0924 09:55:54.305005 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 09:55:54.305650 (+ 645us) server_negotiation.cc:407] Connection header received
0924 09:55:54.305898 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:55:54.305906 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:55:54.306144 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:55:54.306296 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 09:55:54.307012 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:55:54.308412 (+ 1400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:55:54.310347 (+ 1935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:55:54.311501 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:55:54.311614 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:55:54.311779 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0924 09:55:54.312324 (+ 545us) server_negotiation.cc:299] Negotiation successful
0924 09:55:54.312477 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":247,"threads_started":1}
I20260924 09:55:54.312561 2640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:55:54.300715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:42817 (local address 127.0.0.1:44566)
0924 09:55:54.303859 (+ 3144us) negotiation.cc:107] Waiting for socket to connect
0924 09:55:54.303903 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 09:55:54.304153 (+ 250us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:55:54.305066 (+ 913us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:55:54.305075 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:55:54.305101 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 09:55:54.305633 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:55:54.305645 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:55:54.307063 (+ 1418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:55:54.307067 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:55:54.308978 (+ 1911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:55:54.309003 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:55:54.310743 (+ 1740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:55:54.310749 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:55:54.310928 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:55:54.310977 (+ 49us) client_negotiation.cc:769] Sending connection context
0924 09:55:54.311121 (+ 144us) client_negotiation.cc:240] Negotiation successful
0924 09:55:54.311333 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2879,"mutex_wait_us":1,"spinlock_wait_cycles":3456,"thread_start_us":2764,"threads_started":1}
I20260924 09:55:54.313323 2643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:55:54.304292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:35579 (local address 127.0.0.1:34976)
0924 09:55:54.305534 (+ 1242us) negotiation.cc:107] Waiting for socket to connect
0924 09:55:54.305557 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 09:55:54.305738 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:55:54.306501 (+ 763us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:55:54.306506 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:55:54.306528 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 09:55:54.306840 (+ 312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:55:54.306850 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:55:54.308577 (+ 1727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:55:54.308583 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:55:54.310209 (+ 1626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:55:54.310220 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:55:54.311939 (+ 1719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:55:54.311944 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:55:54.312109 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:55:54.312154 (+ 45us) client_negotiation.cc:769] Sending connection context
0924 09:55:54.312885 (+ 731us) client_negotiation.cc:240] Negotiation successful
0924 09:55:54.313006 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1083,"mutex_wait_us":3,"thread_start_us":314,"threads_started":1}
I20260924 09:55:54.316342 2277 tablet_service.cc:1467] Processing CreateTablet for tablet 2c9e5021ba7d44168c03f0e2f4ac71db (DEFAULT_TABLE table=test-table [id=7aa9c068a8d04c8b925362eb8aeb1f4e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 09:55:54.316484 2555 tablet_service.cc:1467] Processing CreateTablet for tablet 2c9e5021ba7d44168c03f0e2f4ac71db (DEFAULT_TABLE table=test-table [id=7aa9c068a8d04c8b925362eb8aeb1f4e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 09:55:54.316959 2416 tablet_service.cc:1467] Processing CreateTablet for tablet 2c9e5021ba7d44168c03f0e2f4ac71db (DEFAULT_TABLE table=test-table [id=7aa9c068a8d04c8b925362eb8aeb1f4e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 09:55:54.317701 2277 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c9e5021ba7d44168c03f0e2f4ac71db. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 09:55:54.317821 2555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c9e5021ba7d44168c03f0e2f4ac71db. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 09:55:54.318112 2416 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c9e5021ba7d44168c03f0e2f4ac71db. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 09:55:54.329658 2644 tablet_bootstrap.cc:492] T 2c9e5021ba7d44168c03f0e2f4ac71db P 9ad27677628b497bb0f4e8636501851d: Bootstrap starting.
I20260924 09:55:54.332155 2645 tablet_bootstrap.cc:492] T 2c9e5021ba7d44168c03f0e2f4ac71db P ccac0f0c5aa54f1abe1d585545d2c34a: Bootstrap starting.
I20260924 09:55:54.332871 2646 tablet_bootstrap.cc:492] T 2c9e5021ba7d44168c03f0e2f4ac71db P e4ce48a623fa4741bed2e12aa3b7a452: Bootstrap starting.
I20260924 09:55:54.333550 2644 tablet_bootstrap.cc:654] T 2c9e5021ba7d44168c03f0e2f4ac71db P 9ad27677628b497bb0f4e8636501851d: Neither blocks nor log segments found. Creating new log.
I20260924 09:55:54.334865 2644 log.cc:824] T 2c9e5021ba7d44168c03f0e2f4ac71db P 9ad27677628b497bb0f4e8636501851d: Log is configured to *not* fsync() on all Append() calls
I20260924 09:55:54.335549 2645 tablet_bootstrap.cc:654] T 2c9e5021ba7d44168c03f0e2f4ac71db P ccac0f0c5aa54f1abe1d585545d2c34a: Neither blocks nor log segments found. Creating new log.
I20260924 09:55:54.336211 2646 tablet_bootstrap.cc:654] T 2c9e5021ba7d44168c03f0e2f4ac71db P e4ce48a623fa4741bed2e12aa3b7a452: Neither blocks nor log segments found. Creating new log.
I20260924 09:55:54.336771 2645 log.cc:824] T 2c9e5021ba7d44168c03f0e2f4ac71db P ccac0f0c5aa54f1abe1d585545d2c34a: Log is configured to *not* fsync() on all Append() calls
I20260924 09:55:54.337641 2646 log.cc:824] T 2c9e5021ba7d44168c03f0e2f4ac71db P e4ce48a623fa4741bed2e12aa3b7a452: Log is configured to *not* fsync() on all Append() calls
I20260924 09:55:54.339828 2644 tablet_bootstrap.cc:492] T 2c9e5021ba7d44168c03f0e2f4ac71db P 9ad27677628b497bb0f4e8636501851d: No bootstrap required, opened a new log
I20260924 09:55:54.339890 2645 tablet_bootstrap.cc:492] T 2c9e5021ba7d44168c03f0e2f4ac71db P ccac0f0c5aa54f1abe1d585545d2c34a: No bootstrap required, opened a new log
I20260924 09:55:54.340124 2644 ts_tablet_manager.cc:1397] T 2c9e5021ba7d44168c03f0e2f4ac71db P 9ad27677628b497bb0f4e8636501851d: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.004s
I20260924 09:55:54.340199 2645 ts_tablet_manager.cc:1397] T 2c9e5021ba7d44168c03f0e2f4ac71db P ccac0f0c5aa54f1abe1d585545d2c34a: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.005s
I20260924 09:55:54.340523 2646 tablet_bootstrap.cc:492] T 2c9e5021ba7d44168c03f0e2f4ac71db P e4ce48a623fa4741bed2e12aa3b7a452: No bootstrap required, opened a new log
I20260924 09:55:54.340767 2646 ts_tablet_manager.cc:1397] T 2c9e5021ba7d44168c03f0e2f4ac71db P e4ce48a623fa4741bed2e12aa3b7a452: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260924 09:55:54.347867 2644 raft_consensus.cc:348] T 2c9e5021ba7d44168c03f0e2f4ac71db P 9ad27677628b497bb0f4e8636501851d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4ce48a623fa4741bed2e12aa3b7a452" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42817 } } peers { permanent_uuid: "9ad27677628b497bb0f4e8636501851d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44953 } } peers { permanent_uuid: "ccac0f0c5aa54f1abe1d585545d2c34a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35579 } }
I20260924 09:55:54.348279 2644 raft_consensus.cc:374] T 2c9e5021ba7d44168c03f0e2f4ac71db P 9ad27677628b497bb0f4e8636501851d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 09:55:54.348403 2644 raft_consensus.cc:729] T 2c9e5021ba7d44168c03f0e2f4ac71db P 9ad27677628b497bb0f4e8636501851d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ad27677628b497bb0f4e8636501851d, State: Initialized, Role: FOLLOWER
I20260924 09:55:54.349022 2644 consensus_queue.cc:260] T 2c9e5021ba7d44168c03f0e2f4ac71db P 9ad27677628b497bb0f4e8636501851d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4ce48a623fa4741bed2e12aa3b7a452" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42817 } } peers { permanent_uuid: "9ad27677628b497bb0f4e8636501851d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44953 } } peers { permanent_uuid: "ccac0f0c5aa54f1abe1d585545d2c34a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35579 } }
I20260924 09:55:54.350778 2644 ts_tablet_manager.cc:1428] T 2c9e5021ba7d44168c03f0e2f4ac71db P 9ad27677628b497bb0f4e8636501851d: Time spent starting tablet: real 0.011s user 0.010s sys 0.000s
I20260924 09:55:54.350720 2646 raft_consensus.cc:348] T 2c9e5021ba7d44168c03f0e2f4ac71db P e4ce48a623fa4741bed2e12aa3b7a452 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4ce48a623fa4741bed2e12aa3b7a452" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42817 } } peers { permanent_uuid: "9ad27677628b497bb0f4e8636501851d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44953 } } peers { permanent_uuid: "ccac0f0c5aa54f1abe1d585545d2c34a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35579 } }
I20260924 09:55:54.350615 2645 raft_consensus.cc:348] T 2c9e5021ba7d44168c03f0e2f4ac71db P ccac0f0c5aa54f1abe1d585545d2c34a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4ce48a623fa4741bed2e12aa3b7a452" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42817 } } peers { permanent_uuid: "9ad27677628b497bb0f4e8636501851d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44953 } } peers { permanent_uuid: "ccac0f0c5aa54f1abe1d585545d2c34a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35579 } }
I20260924 09:55:54.351085 2645 raft_consensus.cc:374] T 2c9e5021ba7d44168c03f0e2f4ac71db P ccac0f0c5aa54f1abe1d585545d2c34a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 09:55:54.351099 2646 raft_consensus.cc:374] T 2c9e5021ba7d44168c03f0e2f4ac71db P e4ce48a623fa4741bed2e12aa3b7a452 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 09:55:54.351197 2646 raft_consensus.cc:729] T 2c9e5021ba7d44168c03f0e2f4ac71db P e4ce48a623fa4741bed2e12aa3b7a452 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4ce48a623fa4741bed2e12aa3b7a452, State: Initialized, Role: FOLLOWER
I20260924 09:55:54.351274 2645 raft_consensus.cc:729] T 2c9e5021ba7d44168c03f0e2f4ac71db P ccac0f0c5aa54f1abe1d585545d2c34a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ccac0f0c5aa54f1abe1d585545d2c34a, State: Initialized, Role: FOLLOWER
I20260924 09:55:54.351843 2646 consensus_queue.cc:260] T 2c9e5021ba7d44168c03f0e2f4ac71db P e4ce48a623fa4741bed2e12aa3b7a452 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4ce48a623fa4741bed2e12aa3b7a452" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42817 } } peers { permanent_uuid: "9ad27677628b497bb0f4e8636501851d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44953 } } peers { permanent_uuid: "ccac0f0c5aa54f1abe1d585545d2c34a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35579 } }
I20260924 09:55:54.351843 2645 consensus_queue.cc:260] T 2c9e5021ba7d44168c03f0e2f4ac71db P ccac0f0c5aa54f1abe1d585545d2c34a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4ce48a623fa4741bed2e12aa3b7a452" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42817 } } peers { permanent_uuid: "9ad27677628b497bb0f4e8636501851d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44953 } } peers { permanent_uuid: "ccac0f0c5aa54f1abe1d585545d2c34a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35579 } }
I20260924 09:55:54.352943 2486 heartbeater.cc:503] Master 127.0.97.62:42045 was elected leader, sending a full tablet report...
I20260924 09:55:54.353653 2645 ts_tablet_manager.cc:1428] T 2c9e5021ba7d44168c03f0e2f4ac71db P ccac0f0c5aa54f1abe1d585545d2c34a: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
I20260924 09:55:54.354650 2625 heartbeater.cc:503] Master 127.0.97.62:42045 was elected leader, sending a full tablet report...
I20260924 09:55:54.355334 2646 ts_tablet_manager.cc:1428] T 2c9e5021ba7d44168c03f0e2f4ac71db P e4ce48a623fa4741bed2e12aa3b7a452: Time spent starting tablet: real 0.014s user 0.016s sys 0.000s
I20260924 09:55:54.395465 2650 raft_consensus.cc:482] T 2c9e5021ba7d44168c03f0e2f4ac71db P 9ad27677628b497bb0f4e8636501851d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 09:55:54.395753 2650 raft_consensus.cc:504] T 2c9e5021ba7d44168c03f0e2f4ac71db P 9ad27677628b497bb0f4e8636501851d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4ce48a623fa4741bed2e12aa3b7a452" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42817 } } peers { permanent_uuid: "9ad27677628b497bb0f4e8636501851d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44953 } } peers { permanent_uuid: "ccac0f0c5aa54f1abe1d585545d2c34a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35579 } }
I20260924 09:55:54.397305 2650 leader_election.cc:288] T 2c9e5021ba7d44168c03f0e2f4ac71db P 9ad27677628b497bb0f4e8636501851d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e4ce48a623fa4741bed2e12aa3b7a452 (127.0.97.3:42817), ccac0f0c5aa54f1abe1d585545d2c34a (127.0.97.2:35579)
I20260924 09:55:54.403950 2639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:55:54.397724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:46871 (local address 127.0.97.3:42817)
0924 09:55:54.397946 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0924 09:55:54.397954 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 09:55:54.398306 (+ 352us) server_negotiation.cc:407] Connection header received
0924 09:55:54.398540 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:55:54.398546 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:55:54.398625 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:55:54.398712 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 09:55:54.399367 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:55:54.400225 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:55:54.402073 (+ 1848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:55:54.402973 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:55:54.403079 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:55:54.403162 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0924 09:55:54.403569 (+ 407us) server_negotiation.cc:299] Negotiation successful
0924 09:55:54.403702 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260924 09:55:54.403970 2653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:55:54.397638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:42817 (local address 127.0.97.1:46871)
0924 09:55:54.398194 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0924 09:55:54.398211 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 09:55:54.398360 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:55:54.398832 (+ 472us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:55:54.398838 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:55:54.398868 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 09:55:54.399222 (+ 354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:55:54.399257 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:55:54.400347 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:55:54.400353 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:55:54.401920 (+ 1567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:55:54.401932 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:55:54.403177 (+ 1245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:55:54.403184 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:55:54.403402 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:55:54.403438 (+ 36us) client_negotiation.cc:769] Sending connection context
0924 09:55:54.403525 (+ 87us) client_negotiation.cc:240] Negotiation successful
0924 09:55:54.403664 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":295,"threads_started":1}
I20260924 09:55:54.404309 2642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:55:54.398639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:34137 (local address 127.0.97.2:35579)
0924 09:55:54.398853 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0924 09:55:54.398859 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 09:55:54.399013 (+ 154us) server_negotiation.cc:407] Connection header received
0924 09:55:54.399151 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:55:54.399155 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:55:54.399226 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:55:54.399341 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 09:55:54.399897 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:55:54.400708 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:55:54.402473 (+ 1765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:55:54.403355 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:55:54.403480 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:55:54.403578 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0924 09:55:54.403937 (+ 359us) server_negotiation.cc:299] Negotiation successful
0924 09:55:54.404068 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260924 09:55:54.404955 2654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:55:54.398355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:35579 (local address 127.0.97.1:34137)
0924 09:55:54.398927 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0924 09:55:54.398940 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 09:55:54.399039 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:55:54.399354 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:55:54.399358 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:55:54.399380 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 09:55:54.399739 (+ 359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:55:54.399749 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:55:54.400840 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:55:54.400844 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:55:54.402311 (+ 1467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:55:54.402324 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:55:54.403577 (+ 1253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:55:54.403584 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:55:54.403748 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:55:54.403784 (+ 36us) client_negotiation.cc:769] Sending connection context
0924 09:55:54.404525 (+ 741us) client_negotiation.cc:240] Negotiation successful
0924 09:55:54.404669 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":262,"threads_started":1}
I20260924 09:55:54.405035 2575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c9e5021ba7d44168c03f0e2f4ac71db" candidate_uuid: "9ad27677628b497bb0f4e8636501851d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4ce48a623fa4741bed2e12aa3b7a452" is_pre_election: true
I20260924 09:55:54.405565 2575 raft_consensus.cc:2454] T 2c9e5021ba7d44168c03f0e2f4ac71db P e4ce48a623fa4741bed2e12aa3b7a452 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ad27677628b497bb0f4e8636501851d in term 0.
I20260924 09:55:54.405797 2436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c9e5021ba7d44168c03f0e2f4ac71db" candidate_uuid: "9ad27677628b497bb0f4e8636501851d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ccac0f0c5aa54f1abe1d585545d2c34a" is_pre_election: true
I20260924 09:55:54.406206 2228 leader_election.cc:302] T 2c9e5021ba7d44168c03f0e2f4ac71db P 9ad27677628b497bb0f4e8636501851d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ad27677628b497bb0f4e8636501851d, e4ce48a623fa4741bed2e12aa3b7a452; no voters:
I20260924 09:55:54.406282 2436 raft_consensus.cc:2454] T 2c9e5021ba7d44168c03f0e2f4ac71db P ccac0f0c5aa54f1abe1d585545d2c34a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ad27677628b497bb0f4e8636501851d in term 0.
I20260924 09:55:54.406627 2650 raft_consensus.cc:2781] T 2c9e5021ba7d44168c03f0e2f4ac71db P 9ad27677628b497bb0f4e8636501851d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 09:55:54.406790 2650 raft_consensus.cc:482] T 2c9e5021ba7d44168c03f0e2f4ac71db P 9ad27677628b497bb0f4e8636501851d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 09:55:54.406894 2650 raft_consensus.cc:3037] T 2c9e5021ba7d44168c03f0e2f4ac71db P 9ad27677628b497bb0f4e8636501851d [term 0 FOLLOWER]: Advancing to term 1
I20260924 09:55:54.409056 2650 raft_consensus.cc:504] T 2c9e5021ba7d44168c03f0e2f4ac71db P 9ad27677628b497bb0f4e8636501851d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4ce48a623fa4741bed2e12aa3b7a452" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42817 } } peers { permanent_uuid: "9ad27677628b497bb0f4e8636501851d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44953 } } peers { permanent_uuid: "ccac0f0c5aa54f1abe1d585545d2c34a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35579 } }
I20260924 09:55:54.409595 2650 leader_election.cc:288] T 2c9e5021ba7d44168c03f0e2f4ac71db P 9ad27677628b497bb0f4e8636501851d [CANDIDATE]: Term 1 election: Requested vote from peers e4ce48a623fa4741bed2e12aa3b7a452 (127.0.97.3:42817), ccac0f0c5aa54f1abe1d585545d2c34a (127.0.97.2:35579)
I20260924 09:55:54.410053 2575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c9e5021ba7d44168c03f0e2f4ac71db" candidate_uuid: "9ad27677628b497bb0f4e8636501851d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4ce48a623fa4741bed2e12aa3b7a452"
I20260924 09:55:54.410111 2436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c9e5021ba7d44168c03f0e2f4ac71db" candidate_uuid: "9ad27677628b497bb0f4e8636501851d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ccac0f0c5aa54f1abe1d585545d2c34a"
I20260924 09:55:54.410233 2575 raft_consensus.cc:3037] T 2c9e5021ba7d44168c03f0e2f4ac71db P e4ce48a623fa4741bed2e12aa3b7a452 [term 0 FOLLOWER]: Advancing to term 1
I20260924 09:55:54.410296 2436 raft_consensus.cc:3037] T 2c9e5021ba7d44168c03f0e2f4ac71db P ccac0f0c5aa54f1abe1d585545d2c34a [term 0 FOLLOWER]: Advancing to term 1
I20260924 09:55:54.412470 2575 raft_consensus.cc:2454] T 2c9e5021ba7d44168c03f0e2f4ac71db P e4ce48a623fa4741bed2e12aa3b7a452 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ad27677628b497bb0f4e8636501851d in term 1.
I20260924 09:55:54.412540 2436 raft_consensus.cc:2454] T 2c9e5021ba7d44168c03f0e2f4ac71db P ccac0f0c5aa54f1abe1d585545d2c34a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ad27677628b497bb0f4e8636501851d in term 1.
I20260924 09:55:54.412875 2228 leader_election.cc:302] T 2c9e5021ba7d44168c03f0e2f4ac71db P 9ad27677628b497bb0f4e8636501851d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ad27677628b497bb0f4e8636501851d, e4ce48a623fa4741bed2e12aa3b7a452; no voters:
I20260924 09:55:54.413162 2650 raft_consensus.cc:2781] T 2c9e5021ba7d44168c03f0e2f4ac71db P 9ad27677628b497bb0f4e8636501851d [term 1 FOLLOWER]: Leader election won for term 1
I20260924 09:55:54.413569 2650 raft_consensus.cc:686] T 2c9e5021ba7d44168c03f0e2f4ac71db P 9ad27677628b497bb0f4e8636501851d [term 1 LEADER]: Becoming Leader. State: Replica: 9ad27677628b497bb0f4e8636501851d, State: Running, Role: LEADER
I20260924 09:55:54.414024 2650 consensus_queue.cc:237] T 2c9e5021ba7d44168c03f0e2f4ac71db P 9ad27677628b497bb0f4e8636501851d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4ce48a623fa4741bed2e12aa3b7a452" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42817 } } peers { permanent_uuid: "9ad27677628b497bb0f4e8636501851d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44953 } } peers { permanent_uuid: "ccac0f0c5aa54f1abe1d585545d2c34a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35579 } }
I20260924 09:55:54.418358 2139 catalog_manager.cc:5495] T 2c9e5021ba7d44168c03f0e2f4ac71db P 9ad27677628b497bb0f4e8636501851d reported cstate change: term changed from 0 to 1, leader changed from <none> to 9ad27677628b497bb0f4e8636501851d (127.0.97.1). New cstate: current_term: 1 leader_uuid: "9ad27677628b497bb0f4e8636501851d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4ce48a623fa4741bed2e12aa3b7a452" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9ad27677628b497bb0f4e8636501851d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44953 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ccac0f0c5aa54f1abe1d585545d2c34a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35579 } health_report { overall_health: UNKNOWN } } }
Sep 24 09:55:54 dist-test-slave-w8d2 krb5kdc[2096](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243752, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 09:55:54.469332 2662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:55:54.453569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42045 (local address 127.0.0.1:48864)
0924 09:55:54.454139 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0924 09:55:54.454156 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 09:55:54.454281 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:55:54.454545 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:55:54.454548 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:55:54.455189 (+ 641us) client_negotiation.cc:189] Negotiated authn=SASL
0924 09:55:54.455460 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:55:54.455469 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:55:54.456410 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:55:54.456414 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:55:54.457368 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:55:54.457378 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:55:54.458025 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:55:54.458030 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:55:54.458178 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:55:54.462408 (+ 4230us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 09:55:54.462436 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0924 09:55:54.464584 (+ 2148us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 09:55:54.467198 (+ 2614us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 09:55:54.467210 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 09:55:54.467216 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 09:55:54.467665 (+ 449us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 09:55:54.467962 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 09:55:54.467964 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 09:55:54.467967 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 09:55:54.468105 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 09:55:54.468542 (+ 437us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 09:55:54.468548 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 09:55:54.468700 (+ 152us) client_negotiation.cc:769] Sending connection context
0924 09:55:54.468825 (+ 125us) client_negotiation.cc:240] Negotiation successful
0924 09:55:54.469032 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":316,"threads_started":1}
I20260924 09:55:54.469396 2189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:55:54.453521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48864 (local address 127.0.97.62:42045)
0924 09:55:54.453693 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0924 09:55:54.453697 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 09:55:54.454235 (+ 538us) server_negotiation.cc:407] Connection header received
0924 09:55:54.454371 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:55:54.454375 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:55:54.454435 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:55:54.454515 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:55:54.455602 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:55:54.456309 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:55:54.457481 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:55:54.457895 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:55:54.457982 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:55:54.464800 (+ 6818us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:55:54.464805 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:55:54.464808 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:55:54.464839 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:55:54.467065 (+ 2226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:55:54.467766 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:55:54.467770 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:55:54.467773 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:55:54.467882 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:55:54.468192 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:55:54.468195 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:55:54.468197 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:55:54.468445 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:55:54.468556 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0924 09:55:54.468917 (+ 361us) server_negotiation.cc:299] Negotiation successful
0924 09:55:54.469137 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":48,"server-negotiator.queue_time_us":53}
I20260924 09:55:54.474035 2139 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:48864:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260924 09:55:54.475019 2139 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260924 09:55:54.479498 2626 tablet.cc:2369] T 2c9e5021ba7d44168c03f0e2f4ac71db P e4ce48a623fa4741bed2e12aa3b7a452: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 09:55:54.482039 2416 tablet_service.cc:1467] Processing CreateTablet for tablet a88241185392497796e003a5891c8d78 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=f7ab1e26d4d745fbb67d36d6b492adea]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 09:55:54.481868 2277 tablet_service.cc:1467] Processing CreateTablet for tablet a88241185392497796e003a5891c8d78 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=f7ab1e26d4d745fbb67d36d6b492adea]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 09:55:54.482206 2555 tablet_service.cc:1467] Processing CreateTablet for tablet a88241185392497796e003a5891c8d78 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=f7ab1e26d4d745fbb67d36d6b492adea]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 09:55:54.482823 2416 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a88241185392497796e003a5891c8d78. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 09:55:54.482848 2277 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a88241185392497796e003a5891c8d78. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 09:55:54.482892 2555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a88241185392497796e003a5891c8d78. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 09:55:54.488600 2645 tablet_bootstrap.cc:492] T a88241185392497796e003a5891c8d78 P ccac0f0c5aa54f1abe1d585545d2c34a: Bootstrap starting.
I20260924 09:55:54.489753 2644 tablet_bootstrap.cc:492] T a88241185392497796e003a5891c8d78 P 9ad27677628b497bb0f4e8636501851d: Bootstrap starting.
I20260924 09:55:54.489784 2646 tablet_bootstrap.cc:492] T a88241185392497796e003a5891c8d78 P e4ce48a623fa4741bed2e12aa3b7a452: Bootstrap starting.
I20260924 09:55:54.490051 2645 tablet_bootstrap.cc:654] T a88241185392497796e003a5891c8d78 P ccac0f0c5aa54f1abe1d585545d2c34a: Neither blocks nor log segments found. Creating new log.
I20260924 09:55:54.491014 2644 tablet_bootstrap.cc:654] T a88241185392497796e003a5891c8d78 P 9ad27677628b497bb0f4e8636501851d: Neither blocks nor log segments found. Creating new log.
I20260924 09:55:54.491297 2646 tablet_bootstrap.cc:654] T a88241185392497796e003a5891c8d78 P e4ce48a623fa4741bed2e12aa3b7a452: Neither blocks nor log segments found. Creating new log.
I20260924 09:55:54.493150 2645 tablet_bootstrap.cc:492] T a88241185392497796e003a5891c8d78 P ccac0f0c5aa54f1abe1d585545d2c34a: No bootstrap required, opened a new log
I20260924 09:55:54.493296 2645 ts_tablet_manager.cc:1397] T a88241185392497796e003a5891c8d78 P ccac0f0c5aa54f1abe1d585545d2c34a: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260924 09:55:54.493777 2644 tablet_bootstrap.cc:492] T a88241185392497796e003a5891c8d78 P 9ad27677628b497bb0f4e8636501851d: No bootstrap required, opened a new log
I20260924 09:55:54.493913 2644 ts_tablet_manager.cc:1397] T a88241185392497796e003a5891c8d78 P 9ad27677628b497bb0f4e8636501851d: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260924 09:55:54.493975 2645 raft_consensus.cc:348] T a88241185392497796e003a5891c8d78 P ccac0f0c5aa54f1abe1d585545d2c34a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ad27677628b497bb0f4e8636501851d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44953 } } peers { permanent_uuid: "ccac0f0c5aa54f1abe1d585545d2c34a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35579 } } peers { permanent_uuid: "e4ce48a623fa4741bed2e12aa3b7a452" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42817 } }
I20260924 09:55:54.494181 2646 tablet_bootstrap.cc:492] T a88241185392497796e003a5891c8d78 P e4ce48a623fa4741bed2e12aa3b7a452: No bootstrap required, opened a new log
I20260924 09:55:54.494194 2645 raft_consensus.cc:374] T a88241185392497796e003a5891c8d78 P ccac0f0c5aa54f1abe1d585545d2c34a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 09:55:54.494277 2645 raft_consensus.cc:729] T a88241185392497796e003a5891c8d78 P ccac0f0c5aa54f1abe1d585545d2c34a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ccac0f0c5aa54f1abe1d585545d2c34a, State: Initialized, Role: FOLLOWER
I20260924 09:55:54.494311 2646 ts_tablet_manager.cc:1397] T a88241185392497796e003a5891c8d78 P e4ce48a623fa4741bed2e12aa3b7a452: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260924 09:55:54.494498 2645 consensus_queue.cc:260] T a88241185392497796e003a5891c8d78 P ccac0f0c5aa54f1abe1d585545d2c34a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ad27677628b497bb0f4e8636501851d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44953 } } peers { permanent_uuid: "ccac0f0c5aa54f1abe1d585545d2c34a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35579 } } peers { permanent_uuid: "e4ce48a623fa4741bed2e12aa3b7a452" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42817 } }
I20260924 09:55:54.494655 2644 raft_consensus.cc:348] T a88241185392497796e003a5891c8d78 P 9ad27677628b497bb0f4e8636501851d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ad27677628b497bb0f4e8636501851d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44953 } } peers { permanent_uuid: "ccac0f0c5aa54f1abe1d585545d2c34a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35579 } } peers { permanent_uuid: "e4ce48a623fa4741bed2e12aa3b7a452" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42817 } }
I20260924 09:55:54.494880 2644 raft_consensus.cc:374] T a88241185392497796e003a5891c8d78 P 9ad27677628b497bb0f4e8636501851d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 09:55:54.494967 2644 raft_consensus.cc:729] T a88241185392497796e003a5891c8d78 P 9ad27677628b497bb0f4e8636501851d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ad27677628b497bb0f4e8636501851d, State: Initialized, Role: FOLLOWER
I20260924 09:55:54.494977 2646 raft_consensus.cc:348] T a88241185392497796e003a5891c8d78 P e4ce48a623fa4741bed2e12aa3b7a452 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ad27677628b497bb0f4e8636501851d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44953 } } peers { permanent_uuid: "ccac0f0c5aa54f1abe1d585545d2c34a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35579 } } peers { permanent_uuid: "e4ce48a623fa4741bed2e12aa3b7a452" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42817 } }
I20260924 09:55:54.494904 2651 tablet_replica.cc:439] T a88241185392497796e003a5891c8d78 P ccac0f0c5aa54f1abe1d585545d2c34a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ad27677628b497bb0f4e8636501851d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44953 } } peers { permanent_uuid: "ccac0f0c5aa54f1abe1d585545d2c34a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35579 } } peers { permanent_uuid: "e4ce48a623fa4741bed2e12aa3b7a452" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42817 } } }
I20260924 09:55:54.495213 2646 raft_consensus.cc:374] T a88241185392497796e003a5891c8d78 P e4ce48a623fa4741bed2e12aa3b7a452 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 09:55:54.495294 2651 tablet_replica.cc:442] T a88241185392497796e003a5891c8d78 P ccac0f0c5aa54f1abe1d585545d2c34a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 09:55:54.495162 2644 consensus_queue.cc:260] T a88241185392497796e003a5891c8d78 P 9ad27677628b497bb0f4e8636501851d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ad27677628b497bb0f4e8636501851d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44953 } } peers { permanent_uuid: "ccac0f0c5aa54f1abe1d585545d2c34a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35579 } } peers { permanent_uuid: "e4ce48a623fa4741bed2e12aa3b7a452" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42817 } }
I20260924 09:55:54.495316 2646 raft_consensus.cc:729] T a88241185392497796e003a5891c8d78 P e4ce48a623fa4741bed2e12aa3b7a452 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4ce48a623fa4741bed2e12aa3b7a452, State: Initialized, Role: FOLLOWER
I20260924 09:55:54.495543 2646 consensus_queue.cc:260] T a88241185392497796e003a5891c8d78 P e4ce48a623fa4741bed2e12aa3b7a452 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ad27677628b497bb0f4e8636501851d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44953 } } peers { permanent_uuid: "ccac0f0c5aa54f1abe1d585545d2c34a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35579 } } peers { permanent_uuid: "e4ce48a623fa4741bed2e12aa3b7a452" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42817 } }
I20260924 09:55:54.495726 2644 ts_tablet_manager.cc:1428] T a88241185392497796e003a5891c8d78 P 9ad27677628b497bb0f4e8636501851d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 09:55:54.495757 2645 ts_tablet_manager.cc:1428] T a88241185392497796e003a5891c8d78 P ccac0f0c5aa54f1abe1d585545d2c34a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260924 09:55:54.496086 2650 tablet_replica.cc:439] T a88241185392497796e003a5891c8d78 P 9ad27677628b497bb0f4e8636501851d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ad27677628b497bb0f4e8636501851d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44953 } } peers { permanent_uuid: "ccac0f0c5aa54f1abe1d585545d2c34a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35579 } } peers { permanent_uuid: "e4ce48a623fa4741bed2e12aa3b7a452" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42817 } } }
I20260924 09:55:54.496364 2650 tablet_replica.cc:442] T a88241185392497796e003a5891c8d78 P 9ad27677628b497bb0f4e8636501851d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 09:55:54.496240 2652 tablet_replica.cc:439] T a88241185392497796e003a5891c8d78 P e4ce48a623fa4741bed2e12aa3b7a452: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ad27677628b497bb0f4e8636501851d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44953 } } peers { permanent_uuid: "ccac0f0c5aa54f1abe1d585545d2c34a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35579 } } peers { permanent_uuid: "e4ce48a623fa4741bed2e12aa3b7a452" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42817 } } }
I20260924 09:55:54.496546 2652 tablet_replica.cc:442] T a88241185392497796e003a5891c8d78 P e4ce48a623fa4741bed2e12aa3b7a452: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 09:55:54.497509 2646 ts_tablet_manager.cc:1428] T a88241185392497796e003a5891c8d78 P e4ce48a623fa4741bed2e12aa3b7a452: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
W20260924 09:55:54.505854 2487 tablet.cc:2369] T 2c9e5021ba7d44168c03f0e2f4ac71db P ccac0f0c5aa54f1abe1d585545d2c34a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 09:55:54.522308 2348 tablet.cc:2369] T 2c9e5021ba7d44168c03f0e2f4ac71db P 9ad27677628b497bb0f4e8636501851d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 09:55:54.564762 2651 raft_consensus.cc:482] T a88241185392497796e003a5891c8d78 P ccac0f0c5aa54f1abe1d585545d2c34a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 09:55:54.564868 2650 raft_consensus.cc:482] T a88241185392497796e003a5891c8d78 P 9ad27677628b497bb0f4e8636501851d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 09:55:54.564997 2651 raft_consensus.cc:504] T a88241185392497796e003a5891c8d78 P ccac0f0c5aa54f1abe1d585545d2c34a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ad27677628b497bb0f4e8636501851d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44953 } } peers { permanent_uuid: "ccac0f0c5aa54f1abe1d585545d2c34a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35579 } } peers { permanent_uuid: "e4ce48a623fa4741bed2e12aa3b7a452" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42817 } }
I20260924 09:55:54.565032 2650 raft_consensus.cc:504] T a88241185392497796e003a5891c8d78 P 9ad27677628b497bb0f4e8636501851d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ad27677628b497bb0f4e8636501851d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44953 } } peers { permanent_uuid: "ccac0f0c5aa54f1abe1d585545d2c34a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35579 } } peers { permanent_uuid: "e4ce48a623fa4741bed2e12aa3b7a452" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42817 } }
I20260924 09:55:54.565649 2650 leader_election.cc:288] T a88241185392497796e003a5891c8d78 P 9ad27677628b497bb0f4e8636501851d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ccac0f0c5aa54f1abe1d585545d2c34a (127.0.97.2:35579), e4ce48a623fa4741bed2e12aa3b7a452 (127.0.97.3:42817)
I20260924 09:55:54.566121 2436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a88241185392497796e003a5891c8d78" candidate_uuid: "9ad27677628b497bb0f4e8636501851d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ccac0f0c5aa54f1abe1d585545d2c34a" is_pre_election: true
I20260924 09:55:54.566224 2575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a88241185392497796e003a5891c8d78" candidate_uuid: "9ad27677628b497bb0f4e8636501851d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4ce48a623fa4741bed2e12aa3b7a452" is_pre_election: true
I20260924 09:55:54.566349 2436 raft_consensus.cc:2454] T a88241185392497796e003a5891c8d78 P ccac0f0c5aa54f1abe1d585545d2c34a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ad27677628b497bb0f4e8636501851d in term 0.
I20260924 09:55:54.566452 2575 raft_consensus.cc:2454] T a88241185392497796e003a5891c8d78 P e4ce48a623fa4741bed2e12aa3b7a452 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ad27677628b497bb0f4e8636501851d in term 0.
I20260924 09:55:54.566744 2651 leader_election.cc:288] T a88241185392497796e003a5891c8d78 P ccac0f0c5aa54f1abe1d585545d2c34a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9ad27677628b497bb0f4e8636501851d (127.0.97.1:44953), e4ce48a623fa4741bed2e12aa3b7a452 (127.0.97.3:42817)
I20260924 09:55:54.566763 2228 leader_election.cc:302] T a88241185392497796e003a5891c8d78 P 9ad27677628b497bb0f4e8636501851d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ad27677628b497bb0f4e8636501851d, ccac0f0c5aa54f1abe1d585545d2c34a; no voters:
I20260924 09:55:54.567011 2650 raft_consensus.cc:2781] T a88241185392497796e003a5891c8d78 P 9ad27677628b497bb0f4e8636501851d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 09:55:54.567097 2650 raft_consensus.cc:482] T a88241185392497796e003a5891c8d78 P 9ad27677628b497bb0f4e8636501851d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 09:55:54.567202 2650 raft_consensus.cc:3037] T a88241185392497796e003a5891c8d78 P 9ad27677628b497bb0f4e8636501851d [term 0 FOLLOWER]: Advancing to term 1
I20260924 09:55:54.569937 2650 raft_consensus.cc:504] T a88241185392497796e003a5891c8d78 P 9ad27677628b497bb0f4e8636501851d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ad27677628b497bb0f4e8636501851d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44953 } } peers { permanent_uuid: "ccac0f0c5aa54f1abe1d585545d2c34a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35579 } } peers { permanent_uuid: "e4ce48a623fa4741bed2e12aa3b7a452" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42817 } }
I20260924 09:55:54.570523 2650 leader_election.cc:288] T a88241185392497796e003a5891c8d78 P 9ad27677628b497bb0f4e8636501851d [CANDIDATE]: Term 1 election: Requested vote from peers ccac0f0c5aa54f1abe1d585545d2c34a (127.0.97.2:35579), e4ce48a623fa4741bed2e12aa3b7a452 (127.0.97.3:42817)
I20260924 09:55:54.571044 2575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a88241185392497796e003a5891c8d78" candidate_uuid: "9ad27677628b497bb0f4e8636501851d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4ce48a623fa4741bed2e12aa3b7a452"
I20260924 09:55:54.571059 2436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a88241185392497796e003a5891c8d78" candidate_uuid: "9ad27677628b497bb0f4e8636501851d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ccac0f0c5aa54f1abe1d585545d2c34a"
I20260924 09:55:54.571322 2436 raft_consensus.cc:3037] T a88241185392497796e003a5891c8d78 P ccac0f0c5aa54f1abe1d585545d2c34a [term 0 FOLLOWER]: Advancing to term 1
I20260924 09:55:54.571346 2575 raft_consensus.cc:3037] T a88241185392497796e003a5891c8d78 P e4ce48a623fa4741bed2e12aa3b7a452 [term 0 FOLLOWER]: Advancing to term 1
I20260924 09:55:54.574407 2575 raft_consensus.cc:2454] T a88241185392497796e003a5891c8d78 P e4ce48a623fa4741bed2e12aa3b7a452 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ad27677628b497bb0f4e8636501851d in term 1.
I20260924 09:55:54.574496 2436 raft_consensus.cc:2454] T a88241185392497796e003a5891c8d78 P ccac0f0c5aa54f1abe1d585545d2c34a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ad27677628b497bb0f4e8636501851d in term 1.
I20260924 09:55:54.574813 2228 leader_election.cc:302] T a88241185392497796e003a5891c8d78 P 9ad27677628b497bb0f4e8636501851d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ad27677628b497bb0f4e8636501851d, e4ce48a623fa4741bed2e12aa3b7a452; no voters:
I20260924 09:55:54.575099 2650 raft_consensus.cc:2781] T a88241185392497796e003a5891c8d78 P 9ad27677628b497bb0f4e8636501851d [term 1 FOLLOWER]: Leader election won for term 1
I20260924 09:55:54.575590 2641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:55:54.567033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:37097 (local address 127.0.97.1:44953)
0924 09:55:54.567419 (+ 386us) server_negotiation.cc:206] Beginning negotiation
0924 09:55:54.567425 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 09:55:54.568122 (+ 697us) server_negotiation.cc:407] Connection header received
0924 09:55:54.568373 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:55:54.568378 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:55:54.568481 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:55:54.568582 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 09:55:54.569114 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:55:54.570016 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:55:54.573050 (+ 3034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:55:54.574227 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:55:54.574332 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:55:54.574428 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0924 09:55:54.575188 (+ 760us) server_negotiation.cc:299] Negotiation successful
0924 09:55:54.575349 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":256}
I20260924 09:55:54.575603 2639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:55:54.567431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:60785 (local address 127.0.97.3:42817)
0924 09:55:54.567578 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0924 09:55:54.567583 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 09:55:54.568483 (+ 900us) server_negotiation.cc:407] Connection header received
0924 09:55:54.568662 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:55:54.568667 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:55:54.568732 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:55:54.568784 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 09:55:54.569436 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:55:54.570423 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:55:54.573066 (+ 2643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:55:54.574316 (+ 1250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:55:54.574410 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:55:54.574504 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0924 09:55:54.575190 (+ 686us) server_negotiation.cc:299] Negotiation successful
0924 09:55:54.575347 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 09:55:54.575358 2650 raft_consensus.cc:686] T a88241185392497796e003a5891c8d78 P 9ad27677628b497bb0f4e8636501851d [term 1 LEADER]: Becoming Leader. State: Replica: 9ad27677628b497bb0f4e8636501851d, State: Running, Role: LEADER
I20260924 09:55:54.575897 2667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:55:54.566974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44953 (local address 127.0.97.2:37097)
0924 09:55:54.568025 (+ 1051us) negotiation.cc:107] Waiting for socket to connect
0924 09:55:54.568045 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 09:55:54.568233 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:55:54.568581 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:55:54.568585 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:55:54.568610 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 09:55:54.568936 (+ 326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:55:54.568948 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:55:54.570248 (+ 1300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:55:54.570253 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:55:54.572869 (+ 2616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:55:54.572886 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:55:54.574748 (+ 1862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:55:54.574755 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:55:54.574993 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:55:54.575048 (+ 55us) client_negotiation.cc:769] Sending connection context
0924 09:55:54.575396 (+ 348us) client_negotiation.cc:240] Negotiation successful
0924 09:55:54.575519 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":919,"mutex_wait_us":80,"thread_start_us":1136,"threads_started":1}
I20260924 09:55:54.576272 2666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:55:54.567112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:42817 (local address 127.0.97.2:60785)
0924 09:55:54.568380 (+ 1268us) negotiation.cc:107] Waiting for socket to connect
0924 09:55:54.568396 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 09:55:54.568510 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:55:54.568921 (+ 411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:55:54.568924 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:55:54.568941 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 09:55:54.569287 (+ 346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:55:54.569299 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:55:54.570579 (+ 1280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:55:54.570588 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:55:54.572892 (+ 2304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:55:54.572906 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:55:54.574717 (+ 1811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:55:54.574723 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:55:54.574970 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:55:54.575046 (+ 76us) client_negotiation.cc:769] Sending connection context
0924 09:55:54.575791 (+ 745us) client_negotiation.cc:240] Negotiation successful
0924 09:55:54.575929 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1167,"mutex_wait_us":87,"thread_start_us":613,"threads_started":1}
I20260924 09:55:54.576207 2650 consensus_queue.cc:237] T a88241185392497796e003a5891c8d78 P 9ad27677628b497bb0f4e8636501851d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ad27677628b497bb0f4e8636501851d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44953 } } peers { permanent_uuid: "ccac0f0c5aa54f1abe1d585545d2c34a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35579 } } peers { permanent_uuid: "e4ce48a623fa4741bed2e12aa3b7a452" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42817 } }
I20260924 09:55:54.576772 2575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a88241185392497796e003a5891c8d78" candidate_uuid: "ccac0f0c5aa54f1abe1d585545d2c34a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4ce48a623fa4741bed2e12aa3b7a452" is_pre_election: true
I20260924 09:55:54.576781 2297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a88241185392497796e003a5891c8d78" candidate_uuid: "ccac0f0c5aa54f1abe1d585545d2c34a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ad27677628b497bb0f4e8636501851d" is_pre_election: true
I20260924 09:55:54.577067 2575 raft_consensus.cc:2379] T a88241185392497796e003a5891c8d78 P e4ce48a623fa4741bed2e12aa3b7a452 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ccac0f0c5aa54f1abe1d585545d2c34a in current term 1: Already voted for candidate 9ad27677628b497bb0f4e8636501851d in this term.
I20260924 09:55:54.577788 2655 tablet_replica.cc:439] T a88241185392497796e003a5891c8d78 P 9ad27677628b497bb0f4e8636501851d: TxnStatusTablet state changed. Reason: New leader 9ad27677628b497bb0f4e8636501851d. Latest consensus state: current_term: 1 leader_uuid: "9ad27677628b497bb0f4e8636501851d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ad27677628b497bb0f4e8636501851d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44953 } } peers { permanent_uuid: "ccac0f0c5aa54f1abe1d585545d2c34a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35579 } } peers { permanent_uuid: "e4ce48a623fa4741bed2e12aa3b7a452" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42817 } } }
I20260924 09:55:54.577944 2370 leader_election.cc:302] T a88241185392497796e003a5891c8d78 P ccac0f0c5aa54f1abe1d585545d2c34a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ccac0f0c5aa54f1abe1d585545d2c34a; no voters: 9ad27677628b497bb0f4e8636501851d, e4ce48a623fa4741bed2e12aa3b7a452
I20260924 09:55:54.578126 2655 tablet_replica.cc:442] T a88241185392497796e003a5891c8d78 P 9ad27677628b497bb0f4e8636501851d: This TxnStatusTablet replica's current role is: LEADER
I20260924 09:55:54.578598 2651 raft_consensus.cc:2726] T a88241185392497796e003a5891c8d78 P ccac0f0c5aa54f1abe1d585545d2c34a [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 09:55:54.578974 2669 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260924 09:55:54.579211 2669 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260924 09:55:54.579635 2141 catalog_manager.cc:5495] T a88241185392497796e003a5891c8d78 P 9ad27677628b497bb0f4e8636501851d reported cstate change: term changed from 0 to 1, leader changed from <none> to 9ad27677628b497bb0f4e8636501851d (127.0.97.1). New cstate: current_term: 1 leader_uuid: "9ad27677628b497bb0f4e8636501851d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ad27677628b497bb0f4e8636501851d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44953 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ccac0f0c5aa54f1abe1d585545d2c34a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35579 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4ce48a623fa4741bed2e12aa3b7a452" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42817 } health_report { overall_health: UNKNOWN } } }
I20260924 09:55:54.597375 2662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:55:54.591438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44953 (local address 127.0.0.1:39520)
0924 09:55:54.591677 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0924 09:55:54.591694 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 09:55:54.591835 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:55:54.592291 (+ 456us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:55:54.592294 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:55:54.592303 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 09:55:54.592465 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:55:54.592473 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:55:54.593873 (+ 1400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:55:54.593877 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:55:54.594837 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:55:54.594848 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:55:54.595768 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:55:54.595773 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:55:54.595929 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:55:54.595983 (+ 54us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 09:55:54.596983 (+ 1000us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 09:55:54.596994 (+ 11us) client_negotiation.cc:769] Sending connection context
0924 09:55:54.597038 (+ 44us) client_negotiation.cc:240] Negotiation successful
0924 09:55:54.597147 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260924 09:55:54.597608 2641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:55:54.591516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39520 (local address 127.0.97.1:44953)
0924 09:55:54.591675 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0924 09:55:54.591682 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 09:55:54.591777 (+ 95us) server_negotiation.cc:407] Connection header received
0924 09:55:54.592031 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:55:54.592037 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:55:54.592137 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:55:54.592282 (+ 145us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 09:55:54.592568 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:55:54.593748 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:55:54.594962 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:55:54.595615 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:55:54.595757 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:55:54.596153 (+ 396us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 09:55:54.596871 (+ 718us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 09:55:54.597028 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0924 09:55:54.597074 (+ 46us) server_negotiation.cc:299] Negotiation successful
0924 09:55:54.597152 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260924 09:55:54.604763 2436 raft_consensus.cc:1261] T a88241185392497796e003a5891c8d78 P ccac0f0c5aa54f1abe1d585545d2c34a [term 1 FOLLOWER]: Refusing update from remote peer 9ad27677628b497bb0f4e8636501851d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 09:55:54.604805 2575 raft_consensus.cc:1261] T a88241185392497796e003a5891c8d78 P e4ce48a623fa4741bed2e12aa3b7a452 [term 1 FOLLOWER]: Refusing update from remote peer 9ad27677628b497bb0f4e8636501851d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 09:55:54.605572 2650 consensus_queue.cc:1035] T a88241185392497796e003a5891c8d78 P 9ad27677628b497bb0f4e8636501851d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ccac0f0c5aa54f1abe1d585545d2c34a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 09:55:54.605877 2655 consensus_queue.cc:1035] T a88241185392497796e003a5891c8d78 P 9ad27677628b497bb0f4e8636501851d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4ce48a623fa4741bed2e12aa3b7a452" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 09:55:54.613145 2651 tablet_replica.cc:439] T a88241185392497796e003a5891c8d78 P ccac0f0c5aa54f1abe1d585545d2c34a: TxnStatusTablet state changed. Reason: New leader 9ad27677628b497bb0f4e8636501851d. Latest consensus state: current_term: 1 leader_uuid: "9ad27677628b497bb0f4e8636501851d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ad27677628b497bb0f4e8636501851d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44953 } } peers { permanent_uuid: "ccac0f0c5aa54f1abe1d585545d2c34a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35579 } } peers { permanent_uuid: "e4ce48a623fa4741bed2e12aa3b7a452" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42817 } } }
I20260924 09:55:54.613441 2651 tablet_replica.cc:442] T a88241185392497796e003a5891c8d78 P ccac0f0c5aa54f1abe1d585545d2c34a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 09:55:54.616324 2673 mvcc.cc:205] Tried to move back new op lower bound from 7332838418851794944 to 7332838418746224640. Current Snapshot: MvccSnapshot[applied={T|T < 7332838418851794944}]
I20260924 09:55:54.619643 2672 mvcc.cc:205] Tried to move back new op lower bound from 7332838418851794944 to 7332838418746224640. Current Snapshot: MvccSnapshot[applied={T|T < 7332838418851794944}]
I20260924 09:55:54.620229 2650 tablet_replica.cc:439] T a88241185392497796e003a5891c8d78 P 9ad27677628b497bb0f4e8636501851d: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9ad27677628b497bb0f4e8636501851d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ad27677628b497bb0f4e8636501851d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44953 } } peers { permanent_uuid: "ccac0f0c5aa54f1abe1d585545d2c34a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35579 } } peers { permanent_uuid: "e4ce48a623fa4741bed2e12aa3b7a452" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42817 } } }
I20260924 09:55:54.620510 2650 tablet_replica.cc:442] T a88241185392497796e003a5891c8d78 P 9ad27677628b497bb0f4e8636501851d: This TxnStatusTablet replica's current role is: LEADER
I20260924 09:55:54.624892 2652 tablet_replica.cc:439] T a88241185392497796e003a5891c8d78 P e4ce48a623fa4741bed2e12aa3b7a452: TxnStatusTablet state changed. Reason: New leader 9ad27677628b497bb0f4e8636501851d. Latest consensus state: current_term: 1 leader_uuid: "9ad27677628b497bb0f4e8636501851d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ad27677628b497bb0f4e8636501851d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44953 } } peers { permanent_uuid: "ccac0f0c5aa54f1abe1d585545d2c34a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35579 } } peers { permanent_uuid: "e4ce48a623fa4741bed2e12aa3b7a452" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42817 } } }
I20260924 09:55:54.625164 2652 tablet_replica.cc:442] T a88241185392497796e003a5891c8d78 P e4ce48a623fa4741bed2e12aa3b7a452: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 09:55:54.627434 2651 tablet_replica.cc:439] T a88241185392497796e003a5891c8d78 P ccac0f0c5aa54f1abe1d585545d2c34a: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9ad27677628b497bb0f4e8636501851d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ad27677628b497bb0f4e8636501851d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44953 } } peers { permanent_uuid: "ccac0f0c5aa54f1abe1d585545d2c34a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35579 } } peers { permanent_uuid: "e4ce48a623fa4741bed2e12aa3b7a452" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42817 } } }
I20260924 09:55:54.627687 2651 tablet_replica.cc:442] T a88241185392497796e003a5891c8d78 P ccac0f0c5aa54f1abe1d585545d2c34a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 09:55:54.629027 2655 tablet_replica.cc:439] T a88241185392497796e003a5891c8d78 P 9ad27677628b497bb0f4e8636501851d: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9ad27677628b497bb0f4e8636501851d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ad27677628b497bb0f4e8636501851d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44953 } } peers { permanent_uuid: "ccac0f0c5aa54f1abe1d585545d2c34a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35579 } } peers { permanent_uuid: "e4ce48a623fa4741bed2e12aa3b7a452" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42817 } } }
I20260924 09:55:54.629283 2655 tablet_replica.cc:442] T a88241185392497796e003a5891c8d78 P 9ad27677628b497bb0f4e8636501851d: This TxnStatusTablet replica's current role is: LEADER
I20260924 09:55:54.638373 2652 tablet_replica.cc:439] T a88241185392497796e003a5891c8d78 P e4ce48a623fa4741bed2e12aa3b7a452: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9ad27677628b497bb0f4e8636501851d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ad27677628b497bb0f4e8636501851d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44953 } } peers { permanent_uuid: "ccac0f0c5aa54f1abe1d585545d2c34a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35579 } } peers { permanent_uuid: "e4ce48a623fa4741bed2e12aa3b7a452" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42817 } } }
I20260924 09:55:54.638625 2652 tablet_replica.cc:442] T a88241185392497796e003a5891c8d78 P e4ce48a623fa4741bed2e12aa3b7a452: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 09:55:54.644383 2641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:55:54.639460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39524 (local address 127.0.97.1:44953)
0924 09:55:54.639784 (+ 324us) server_negotiation.cc:206] Beginning negotiation
0924 09:55:54.639789 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 09:55:54.639805 (+ 16us) server_negotiation.cc:407] Connection header received
0924 09:55:54.639885 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:55:54.639890 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:55:54.639972 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:55:54.640072 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 09:55:54.640516 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:55:54.641252 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:55:54.642538 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:55:54.643199 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:55:54.643345 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:55:54.643710 (+ 365us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 09:55:54.643840 (+ 130us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 09:55:54.643929 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0924 09:55:54.644006 (+ 77us) server_negotiation.cc:299] Negotiation successful
0924 09:55:54.644089 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":214}
I20260924 09:55:54.661190 2641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:55:54.655026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:33899 (local address 127.0.97.1:44953)
0924 09:55:54.655210 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0924 09:55:54.655215 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 09:55:54.655906 (+ 691us) server_negotiation.cc:407] Connection header received
0924 09:55:54.656089 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:55:54.656092 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:55:54.656152 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:55:54.656223 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 09:55:54.656727 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:55:54.657498 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:55:54.659049 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:55:54.659802 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:55:54.659943 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:55:54.660368 (+ 425us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 09:55:54.660522 (+ 154us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 09:55:54.660615 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0924 09:55:54.660742 (+ 127us) server_negotiation.cc:299] Negotiation successful
0924 09:55:54.660821 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 09:55:54.661218 2684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:55:54.655106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44953 (local address 127.0.97.1:33899)
0924 09:55:54.655802 (+ 696us) negotiation.cc:107] Waiting for socket to connect
0924 09:55:54.655819 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 09:55:54.655964 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:55:54.656244 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:55:54.656248 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:55:54.656258 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 09:55:54.656590 (+ 332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:55:54.656600 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:55:54.657633 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:55:54.657638 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:55:54.658891 (+ 1253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:55:54.658905 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:55:54.659992 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:55:54.659997 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:55:54.660120 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:55:54.660183 (+ 63us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 09:55:54.660636 (+ 453us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 09:55:54.660651 (+ 15us) client_negotiation.cc:769] Sending connection context
0924 09:55:54.660708 (+ 57us) client_negotiation.cc:240] Negotiation successful
0924 09:55:54.660818 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":555,"thread_start_us":403,"threads_started":1}
I20260924 09:55:54.670105 2575 raft_consensus.cc:1261] T 2c9e5021ba7d44168c03f0e2f4ac71db P e4ce48a623fa4741bed2e12aa3b7a452 [term 1 FOLLOWER]: Refusing update from remote peer 9ad27677628b497bb0f4e8636501851d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 09:55:54.670630 2436 raft_consensus.cc:1261] T 2c9e5021ba7d44168c03f0e2f4ac71db P ccac0f0c5aa54f1abe1d585545d2c34a [term 1 FOLLOWER]: Refusing update from remote peer 9ad27677628b497bb0f4e8636501851d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 09:55:54.670620 2680 consensus_queue.cc:1035] T 2c9e5021ba7d44168c03f0e2f4ac71db P 9ad27677628b497bb0f4e8636501851d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4ce48a623fa4741bed2e12aa3b7a452" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 09:55:54.671156 2650 consensus_queue.cc:1035] T 2c9e5021ba7d44168c03f0e2f4ac71db P 9ad27677628b497bb0f4e8636501851d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ccac0f0c5aa54f1abe1d585545d2c34a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 09:55:54.675654 2674 mvcc.cc:205] Tried to move back new op lower bound from 7332838419124109312 to 7332838418083201024. Current Snapshot: MvccSnapshot[applied={T|T < 7332838419124109312}]
I20260924 09:55:54.751489 2139 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48854:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 09:55:54.751740 2139 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48854:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 09:55:54.755188 2139 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fcbcd495a6484513824d800e6faedb15: Sending DeleteTablet for 3 replicas of tablet 2c9e5021ba7d44168c03f0e2f4ac71db
I20260924 09:55:54.756193 2555 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c9e5021ba7d44168c03f0e2f4ac71db with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 09:55:54 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:44566
I20260924 09:55:54.756811 2277 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c9e5021ba7d44168c03f0e2f4ac71db with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 09:55:54 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:39514
I20260924 09:55:54.756925 2690 tablet_replica.cc:330] T 2c9e5021ba7d44168c03f0e2f4ac71db P e4ce48a623fa4741bed2e12aa3b7a452: stopping tablet replica
I20260924 09:55:54.757470 2690 raft_consensus.cc:2229] T 2c9e5021ba7d44168c03f0e2f4ac71db P e4ce48a623fa4741bed2e12aa3b7a452 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 09:55:54.757862 2690 raft_consensus.cc:2258] T 2c9e5021ba7d44168c03f0e2f4ac71db P e4ce48a623fa4741bed2e12aa3b7a452 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 09:55:54.758203 2416 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c9e5021ba7d44168c03f0e2f4ac71db with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 09:55:54 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:34976
I20260924 09:55:54.760200 2690 ts_tablet_manager.cc:1905] T 2c9e5021ba7d44168c03f0e2f4ac71db P e4ce48a623fa4741bed2e12aa3b7a452: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 09:55:54.762717 2692 tablet_replica.cc:330] T 2c9e5021ba7d44168c03f0e2f4ac71db P ccac0f0c5aa54f1abe1d585545d2c34a: stopping tablet replica
I20260924 09:55:54.764377 2141 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48854:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 09:55:54.765039 2141 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 09:55:54.765281 2692 raft_consensus.cc:2229] T 2c9e5021ba7d44168c03f0e2f4ac71db P ccac0f0c5aa54f1abe1d585545d2c34a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 09:55:54.765767 2692 raft_consensus.cc:2258] T 2c9e5021ba7d44168c03f0e2f4ac71db P ccac0f0c5aa54f1abe1d585545d2c34a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 09:55:54.770020 2690 ts_tablet_manager.cc:1918] T 2c9e5021ba7d44168c03f0e2f4ac71db P e4ce48a623fa4741bed2e12aa3b7a452: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 09:55:54.771552 2692 ts_tablet_manager.cc:1905] T 2c9e5021ba7d44168c03f0e2f4ac71db P ccac0f0c5aa54f1abe1d585545d2c34a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 09:55:54.775115 2276 tablet_service.cc:1467] Processing CreateTablet for tablet 00191cf27d344f3d9eee0a6a5d84b5f7 (DEFAULT_TABLE table=test-table [id=3b2bf598588c4962a124ec1096b68a89]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 09:55:54.775770 2276 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00191cf27d344f3d9eee0a6a5d84b5f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 09:55:54.776835 2416 tablet_service.cc:1467] Processing CreateTablet for tablet 00191cf27d344f3d9eee0a6a5d84b5f7 (DEFAULT_TABLE table=test-table [id=3b2bf598588c4962a124ec1096b68a89]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 09:55:54.777235 2416 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00191cf27d344f3d9eee0a6a5d84b5f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 09:55:54.782456 2644 tablet_bootstrap.cc:492] T 00191cf27d344f3d9eee0a6a5d84b5f7 P 9ad27677628b497bb0f4e8636501851d: Bootstrap starting.
I20260924 09:55:54.784291 2644 tablet_bootstrap.cc:654] T 00191cf27d344f3d9eee0a6a5d84b5f7 P 9ad27677628b497bb0f4e8636501851d: Neither blocks nor log segments found. Creating new log.
I20260924 09:55:54.787817 2645 tablet_bootstrap.cc:492] T 00191cf27d344f3d9eee0a6a5d84b5f7 P ccac0f0c5aa54f1abe1d585545d2c34a: Bootstrap starting.
I20260924 09:55:54.789078 2690 log.cc:1196] T 2c9e5021ba7d44168c03f0e2f4ac71db P e4ce48a623fa4741bed2e12aa3b7a452: Deleting WAL directory at /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790243735634835-388-0/minicluster-data/ts-2/wal/wals/2c9e5021ba7d44168c03f0e2f4ac71db
I20260924 09:55:54.789516 2645 tablet_bootstrap.cc:654] T 00191cf27d344f3d9eee0a6a5d84b5f7 P ccac0f0c5aa54f1abe1d585545d2c34a: Neither blocks nor log segments found. Creating new log.
I20260924 09:55:54.790141 2690 ts_tablet_manager.cc:1939] T 2c9e5021ba7d44168c03f0e2f4ac71db P e4ce48a623fa4741bed2e12aa3b7a452: Deleting consensus metadata
I20260924 09:55:54.775641 2555 tablet_service.cc:1467] Processing CreateTablet for tablet 00191cf27d344f3d9eee0a6a5d84b5f7 (DEFAULT_TABLE table=test-table [id=3b2bf598588c4962a124ec1096b68a89]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 09:55:54.792131 2691 tablet_replica.cc:330] T 2c9e5021ba7d44168c03f0e2f4ac71db P 9ad27677628b497bb0f4e8636501851d: stopping tablet replica
I20260924 09:55:54.792214 2555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00191cf27d344f3d9eee0a6a5d84b5f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 09:55:54.792711 2691 raft_consensus.cc:2229] T 2c9e5021ba7d44168c03f0e2f4ac71db P 9ad27677628b497bb0f4e8636501851d [term 1 LEADER]: Raft consensus shutting down.
I20260924 09:55:54.793346 2691 raft_consensus.cc:2258] T 2c9e5021ba7d44168c03f0e2f4ac71db P 9ad27677628b497bb0f4e8636501851d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 09:55:54.795531 2125 catalog_manager.cc:4841] TS e4ce48a623fa4741bed2e12aa3b7a452 (127.0.97.3:42817): tablet 2c9e5021ba7d44168c03f0e2f4ac71db (table test-table [id=7aa9c068a8d04c8b925362eb8aeb1f4e]) successfully deleted
I20260924 09:55:54.798568 2646 tablet_bootstrap.cc:492] T 00191cf27d344f3d9eee0a6a5d84b5f7 P e4ce48a623fa4741bed2e12aa3b7a452: Bootstrap starting.
I20260924 09:55:54.799096 2691 ts_tablet_manager.cc:1905] T 2c9e5021ba7d44168c03f0e2f4ac71db P 9ad27677628b497bb0f4e8636501851d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 09:55:54.800232 2646 tablet_bootstrap.cc:654] T 00191cf27d344f3d9eee0a6a5d84b5f7 P e4ce48a623fa4741bed2e12aa3b7a452: Neither blocks nor log segments found. Creating new log.
I20260924 09:55:54.802299 2692 ts_tablet_manager.cc:1918] T 2c9e5021ba7d44168c03f0e2f4ac71db P ccac0f0c5aa54f1abe1d585545d2c34a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 09:55:54.803531 2692 log.cc:1196] T 2c9e5021ba7d44168c03f0e2f4ac71db P ccac0f0c5aa54f1abe1d585545d2c34a: Deleting WAL directory at /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790243735634835-388-0/minicluster-data/ts-1/wal/wals/2c9e5021ba7d44168c03f0e2f4ac71db
I20260924 09:55:54.804232 2692 ts_tablet_manager.cc:1939] T 2c9e5021ba7d44168c03f0e2f4ac71db P ccac0f0c5aa54f1abe1d585545d2c34a: Deleting consensus metadata
I20260924 09:55:54.806881 2125 catalog_manager.cc:4841] TS ccac0f0c5aa54f1abe1d585545d2c34a (127.0.97.2:35579): tablet 2c9e5021ba7d44168c03f0e2f4ac71db (table test-table [id=7aa9c068a8d04c8b925362eb8aeb1f4e]) successfully deleted
I20260924 09:55:54.809856 2646 tablet_bootstrap.cc:492] T 00191cf27d344f3d9eee0a6a5d84b5f7 P e4ce48a623fa4741bed2e12aa3b7a452: No bootstrap required, opened a new log
I20260924 09:55:54.810068 2645 tablet_bootstrap.cc:492] T 00191cf27d344f3d9eee0a6a5d84b5f7 P ccac0f0c5aa54f1abe1d585545d2c34a: No bootstrap required, opened a new log
I20260924 09:55:54.810071 2646 ts_tablet_manager.cc:1397] T 00191cf27d344f3d9eee0a6a5d84b5f7 P e4ce48a623fa4741bed2e12aa3b7a452: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.004s
I20260924 09:55:54.810225 2645 ts_tablet_manager.cc:1397] T 00191cf27d344f3d9eee0a6a5d84b5f7 P ccac0f0c5aa54f1abe1d585545d2c34a: Time spent bootstrapping tablet: real 0.022s user 0.007s sys 0.002s
I20260924 09:55:54.810909 2646 raft_consensus.cc:348] T 00191cf27d344f3d9eee0a6a5d84b5f7 P e4ce48a623fa4741bed2e12aa3b7a452 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ad27677628b497bb0f4e8636501851d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44953 } } peers { permanent_uuid: "ccac0f0c5aa54f1abe1d585545d2c34a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35579 } } peers { permanent_uuid: "e4ce48a623fa4741bed2e12aa3b7a452" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42817 } }
I20260924 09:55:54.810907 2645 raft_consensus.cc:348] T 00191cf27d344f3d9eee0a6a5d84b5f7 P ccac0f0c5aa54f1abe1d585545d2c34a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ad27677628b497bb0f4e8636501851d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44953 } } peers { permanent_uuid: "ccac0f0c5aa54f1abe1d585545d2c34a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35579 } } peers { permanent_uuid: "e4ce48a623fa4741bed2e12aa3b7a452" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42817 } }
I20260924 09:55:54.811141 2645 raft_consensus.cc:374] T 00191cf27d344f3d9eee0a6a5d84b5f7 P ccac0f0c5aa54f1abe1d585545d2c34a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 09:55:54.811216 2645 raft_consensus.cc:729] T 00191cf27d344f3d9eee0a6a5d84b5f7 P ccac0f0c5aa54f1abe1d585545d2c34a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ccac0f0c5aa54f1abe1d585545d2c34a, State: Initialized, Role: FOLLOWER
I20260924 09:55:54.811295 2646 raft_consensus.cc:374] T 00191cf27d344f3d9eee0a6a5d84b5f7 P e4ce48a623fa4741bed2e12aa3b7a452 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 09:55:54.811450 2646 raft_consensus.cc:729] T 00191cf27d344f3d9eee0a6a5d84b5f7 P e4ce48a623fa4741bed2e12aa3b7a452 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4ce48a623fa4741bed2e12aa3b7a452, State: Initialized, Role: FOLLOWER
I20260924 09:55:54.811512 2645 consensus_queue.cc:260] T 00191cf27d344f3d9eee0a6a5d84b5f7 P ccac0f0c5aa54f1abe1d585545d2c34a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ad27677628b497bb0f4e8636501851d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44953 } } peers { permanent_uuid: "ccac0f0c5aa54f1abe1d585545d2c34a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35579 } } peers { permanent_uuid: "e4ce48a623fa4741bed2e12aa3b7a452" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42817 } }
I20260924 09:55:54.811848 2646 consensus_queue.cc:260] T 00191cf27d344f3d9eee0a6a5d84b5f7 P e4ce48a623fa4741bed2e12aa3b7a452 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ad27677628b497bb0f4e8636501851d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44953 } } peers { permanent_uuid: "ccac0f0c5aa54f1abe1d585545d2c34a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35579 } } peers { permanent_uuid: "e4ce48a623fa4741bed2e12aa3b7a452" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42817 } }
I20260924 09:55:54.813169 2691 ts_tablet_manager.cc:1918] T 2c9e5021ba7d44168c03f0e2f4ac71db P 9ad27677628b497bb0f4e8636501851d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 09:55:54.813360 2691 log.cc:1196] T 2c9e5021ba7d44168c03f0e2f4ac71db P 9ad27677628b497bb0f4e8636501851d: Deleting WAL directory at /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790243735634835-388-0/minicluster-data/ts-0/wal/wals/2c9e5021ba7d44168c03f0e2f4ac71db
I20260924 09:55:54.814175 2691 ts_tablet_manager.cc:1939] T 2c9e5021ba7d44168c03f0e2f4ac71db P 9ad27677628b497bb0f4e8636501851d: Deleting consensus metadata
I20260924 09:55:54.813741 2645 ts_tablet_manager.cc:1428] T 00191cf27d344f3d9eee0a6a5d84b5f7 P ccac0f0c5aa54f1abe1d585545d2c34a: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260924 09:55:54.815745 2644 tablet_bootstrap.cc:492] T 00191cf27d344f3d9eee0a6a5d84b5f7 P 9ad27677628b497bb0f4e8636501851d: No bootstrap required, opened a new log
I20260924 09:55:54.815904 2644 ts_tablet_manager.cc:1397] T 00191cf27d344f3d9eee0a6a5d84b5f7 P 9ad27677628b497bb0f4e8636501851d: Time spent bootstrapping tablet: real 0.033s user 0.004s sys 0.011s
I20260924 09:55:54.816108 2128 catalog_manager.cc:4841] TS 9ad27677628b497bb0f4e8636501851d (127.0.97.1:44953): tablet 2c9e5021ba7d44168c03f0e2f4ac71db (table test-table [id=7aa9c068a8d04c8b925362eb8aeb1f4e]) successfully deleted
I20260924 09:55:54.816537 2646 ts_tablet_manager.cc:1428] T 00191cf27d344f3d9eee0a6a5d84b5f7 P e4ce48a623fa4741bed2e12aa3b7a452: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20260924 09:55:54.816660 2644 raft_consensus.cc:348] T 00191cf27d344f3d9eee0a6a5d84b5f7 P 9ad27677628b497bb0f4e8636501851d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ad27677628b497bb0f4e8636501851d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44953 } } peers { permanent_uuid: "ccac0f0c5aa54f1abe1d585545d2c34a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35579 } } peers { permanent_uuid: "e4ce48a623fa4741bed2e12aa3b7a452" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42817 } }
I20260924 09:55:54.816905 2644 raft_consensus.cc:374] T 00191cf27d344f3d9eee0a6a5d84b5f7 P 9ad27677628b497bb0f4e8636501851d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 09:55:54.816999 2644 raft_consensus.cc:729] T 00191cf27d344f3d9eee0a6a5d84b5f7 P 9ad27677628b497bb0f4e8636501851d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ad27677628b497bb0f4e8636501851d, State: Initialized, Role: FOLLOWER
I20260924 09:55:54.817248 2644 consensus_queue.cc:260] T 00191cf27d344f3d9eee0a6a5d84b5f7 P 9ad27677628b497bb0f4e8636501851d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ad27677628b497bb0f4e8636501851d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44953 } } peers { permanent_uuid: "ccac0f0c5aa54f1abe1d585545d2c34a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35579 } } peers { permanent_uuid: "e4ce48a623fa4741bed2e12aa3b7a452" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42817 } }
I20260924 09:55:54.817840 2644 ts_tablet_manager.cc:1428] T 00191cf27d344f3d9eee0a6a5d84b5f7 P 9ad27677628b497bb0f4e8636501851d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 09:55:55.137394 2651 raft_consensus.cc:482] T 00191cf27d344f3d9eee0a6a5d84b5f7 P ccac0f0c5aa54f1abe1d585545d2c34a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 09:55:55.137596 2651 raft_consensus.cc:504] T 00191cf27d344f3d9eee0a6a5d84b5f7 P ccac0f0c5aa54f1abe1d585545d2c34a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ad27677628b497bb0f4e8636501851d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44953 } } peers { permanent_uuid: "ccac0f0c5aa54f1abe1d585545d2c34a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35579 } } peers { permanent_uuid: "e4ce48a623fa4741bed2e12aa3b7a452" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42817 } }
I20260924 09:55:55.138280 2651 leader_election.cc:288] T 00191cf27d344f3d9eee0a6a5d84b5f7 P ccac0f0c5aa54f1abe1d585545d2c34a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9ad27677628b497bb0f4e8636501851d (127.0.97.1:44953), e4ce48a623fa4741bed2e12aa3b7a452 (127.0.97.3:42817)
I20260924 09:55:55.138826 2297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00191cf27d344f3d9eee0a6a5d84b5f7" candidate_uuid: "ccac0f0c5aa54f1abe1d585545d2c34a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ad27677628b497bb0f4e8636501851d" is_pre_election: true
I20260924 09:55:55.138895 2574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00191cf27d344f3d9eee0a6a5d84b5f7" candidate_uuid: "ccac0f0c5aa54f1abe1d585545d2c34a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4ce48a623fa4741bed2e12aa3b7a452" is_pre_election: true
I20260924 09:55:55.139101 2574 raft_consensus.cc:2454] T 00191cf27d344f3d9eee0a6a5d84b5f7 P e4ce48a623fa4741bed2e12aa3b7a452 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ccac0f0c5aa54f1abe1d585545d2c34a in term 0.
I20260924 09:55:55.139156 2297 raft_consensus.cc:2454] T 00191cf27d344f3d9eee0a6a5d84b5f7 P 9ad27677628b497bb0f4e8636501851d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ccac0f0c5aa54f1abe1d585545d2c34a in term 0.
I20260924 09:55:55.139653 2367 leader_election.cc:302] T 00191cf27d344f3d9eee0a6a5d84b5f7 P ccac0f0c5aa54f1abe1d585545d2c34a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ccac0f0c5aa54f1abe1d585545d2c34a, e4ce48a623fa4741bed2e12aa3b7a452; no voters:
I20260924 09:55:55.139984 2651 raft_consensus.cc:2781] T 00191cf27d344f3d9eee0a6a5d84b5f7 P ccac0f0c5aa54f1abe1d585545d2c34a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 09:55:55.140089 2651 raft_consensus.cc:482] T 00191cf27d344f3d9eee0a6a5d84b5f7 P ccac0f0c5aa54f1abe1d585545d2c34a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 09:55:55.140182 2651 raft_consensus.cc:3037] T 00191cf27d344f3d9eee0a6a5d84b5f7 P ccac0f0c5aa54f1abe1d585545d2c34a [term 0 FOLLOWER]: Advancing to term 1
I20260924 09:55:55.142117 2651 raft_consensus.cc:504] T 00191cf27d344f3d9eee0a6a5d84b5f7 P ccac0f0c5aa54f1abe1d585545d2c34a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ad27677628b497bb0f4e8636501851d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44953 } } peers { permanent_uuid: "ccac0f0c5aa54f1abe1d585545d2c34a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35579 } } peers { permanent_uuid: "e4ce48a623fa4741bed2e12aa3b7a452" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42817 } }
I20260924 09:55:55.142700 2651 leader_election.cc:288] T 00191cf27d344f3d9eee0a6a5d84b5f7 P ccac0f0c5aa54f1abe1d585545d2c34a [CANDIDATE]: Term 1 election: Requested vote from peers 9ad27677628b497bb0f4e8636501851d (127.0.97.1:44953), e4ce48a623fa4741bed2e12aa3b7a452 (127.0.97.3:42817)
I20260924 09:55:55.143131 2297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00191cf27d344f3d9eee0a6a5d84b5f7" candidate_uuid: "ccac0f0c5aa54f1abe1d585545d2c34a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ad27677628b497bb0f4e8636501851d"
I20260924 09:55:55.143293 2574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00191cf27d344f3d9eee0a6a5d84b5f7" candidate_uuid: "ccac0f0c5aa54f1abe1d585545d2c34a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4ce48a623fa4741bed2e12aa3b7a452"
I20260924 09:55:55.143401 2297 raft_consensus.cc:3037] T 00191cf27d344f3d9eee0a6a5d84b5f7 P 9ad27677628b497bb0f4e8636501851d [term 0 FOLLOWER]: Advancing to term 1
I20260924 09:55:55.143496 2574 raft_consensus.cc:3037] T 00191cf27d344f3d9eee0a6a5d84b5f7 P e4ce48a623fa4741bed2e12aa3b7a452 [term 0 FOLLOWER]: Advancing to term 1
I20260924 09:55:55.146235 2574 raft_consensus.cc:2454] T 00191cf27d344f3d9eee0a6a5d84b5f7 P e4ce48a623fa4741bed2e12aa3b7a452 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ccac0f0c5aa54f1abe1d585545d2c34a in term 1.
I20260924 09:55:55.146394 2297 raft_consensus.cc:2454] T 00191cf27d344f3d9eee0a6a5d84b5f7 P 9ad27677628b497bb0f4e8636501851d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ccac0f0c5aa54f1abe1d585545d2c34a in term 1.
I20260924 09:55:55.146684 2367 leader_election.cc:302] T 00191cf27d344f3d9eee0a6a5d84b5f7 P ccac0f0c5aa54f1abe1d585545d2c34a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ccac0f0c5aa54f1abe1d585545d2c34a, e4ce48a623fa4741bed2e12aa3b7a452; no voters:
I20260924 09:55:55.146957 2651 raft_consensus.cc:2781] T 00191cf27d344f3d9eee0a6a5d84b5f7 P ccac0f0c5aa54f1abe1d585545d2c34a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 09:55:55.147370 2651 raft_consensus.cc:686] T 00191cf27d344f3d9eee0a6a5d84b5f7 P ccac0f0c5aa54f1abe1d585545d2c34a [term 1 LEADER]: Becoming Leader. State: Replica: ccac0f0c5aa54f1abe1d585545d2c34a, State: Running, Role: LEADER
I20260924 09:55:55.147751 2651 consensus_queue.cc:237] T 00191cf27d344f3d9eee0a6a5d84b5f7 P ccac0f0c5aa54f1abe1d585545d2c34a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ad27677628b497bb0f4e8636501851d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44953 } } peers { permanent_uuid: "ccac0f0c5aa54f1abe1d585545d2c34a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35579 } } peers { permanent_uuid: "e4ce48a623fa4741bed2e12aa3b7a452" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42817 } }
I20260924 09:55:55.151119 2141 catalog_manager.cc:5495] T 00191cf27d344f3d9eee0a6a5d84b5f7 P ccac0f0c5aa54f1abe1d585545d2c34a reported cstate change: term changed from 0 to 1, leader changed from <none> to ccac0f0c5aa54f1abe1d585545d2c34a (127.0.97.2). New cstate: current_term: 1 leader_uuid: "ccac0f0c5aa54f1abe1d585545d2c34a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ad27677628b497bb0f4e8636501851d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44953 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ccac0f0c5aa54f1abe1d585545d2c34a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35579 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4ce48a623fa4741bed2e12aa3b7a452" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42817 } health_report { overall_health: UNKNOWN } } }
I20260924 09:55:55.270510 2707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:55:55.262849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34990 (local address 127.0.97.2:35579)
0924 09:55:55.264184 (+ 1335us) server_negotiation.cc:206] Beginning negotiation
0924 09:55:55.264190 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 09:55:55.264209 (+ 19us) server_negotiation.cc:407] Connection header received
0924 09:55:55.264305 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:55:55.264311 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:55:55.264407 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:55:55.264583 (+ 176us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 09:55:55.265034 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:55:55.266088 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:55:55.267887 (+ 1799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:55:55.268506 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:55:55.268633 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:55:55.269104 (+ 471us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 09:55:55.269863 (+ 759us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 09:55:55.270012 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0924 09:55:55.270114 (+ 102us) server_negotiation.cc:299] Negotiation successful
0924 09:55:55.270199 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1213,"thread_start_us":1087,"threads_started":1}
I20260924 09:55:55.304021 2711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:55:55.283580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44953 (local address 127.0.97.2:32821)
0924 09:55:55.296148 (+ 12568us) negotiation.cc:107] Waiting for socket to connect
0924 09:55:55.296175 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0924 09:55:55.296368 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:55:55.298067 (+ 1699us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:55:55.298073 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:55:55.298086 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 09:55:55.298395 (+ 309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:55:55.298407 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:55:55.299970 (+ 1563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:55:55.299977 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:55:55.301586 (+ 1609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:55:55.301603 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:55:55.302636 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:55:55.302642 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:55:55.302828 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:55:55.302912 (+ 84us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 09:55:55.303418 (+ 506us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 09:55:55.303454 (+ 36us) client_negotiation.cc:769] Sending connection context
0924 09:55:55.303530 (+ 76us) client_negotiation.cc:240] Negotiation successful
0924 09:55:55.303669 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12333,"thread_start_us":12160,"threads_started":1}
I20260924 09:55:55.304046 2712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:55:55.283619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:32821 (local address 127.0.97.1:44953)
0924 09:55:55.297648 (+ 14029us) server_negotiation.cc:206] Beginning negotiation
0924 09:55:55.297654 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 09:55:55.297687 (+ 33us) server_negotiation.cc:407] Connection header received
0924 09:55:55.297785 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:55:55.297791 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:55:55.297885 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:55:55.298020 (+ 135us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 09:55:55.298576 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:55:55.299815 (+ 1239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:55:55.301750 (+ 1935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:55:55.302462 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:55:55.302622 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:55:55.303147 (+ 525us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 09:55:55.303320 (+ 173us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 09:55:55.303421 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0924 09:55:55.303588 (+ 167us) server_negotiation.cc:299] Negotiation successful
0924 09:55:55.303680 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13897,"thread_start_us":13707,"threads_started":1}
I20260924 09:55:55.313277 2297 raft_consensus.cc:1261] T 00191cf27d344f3d9eee0a6a5d84b5f7 P 9ad27677628b497bb0f4e8636501851d [term 1 FOLLOWER]: Refusing update from remote peer ccac0f0c5aa54f1abe1d585545d2c34a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 09:55:55.313871 2574 raft_consensus.cc:1261] T 00191cf27d344f3d9eee0a6a5d84b5f7 P e4ce48a623fa4741bed2e12aa3b7a452 [term 1 FOLLOWER]: Refusing update from remote peer ccac0f0c5aa54f1abe1d585545d2c34a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 09:55:55.313948 2696 consensus_queue.cc:1035] T 00191cf27d344f3d9eee0a6a5d84b5f7 P ccac0f0c5aa54f1abe1d585545d2c34a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ad27677628b497bb0f4e8636501851d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 09:55:55.314853 2696 consensus_queue.cc:1035] T 00191cf27d344f3d9eee0a6a5d84b5f7 P ccac0f0c5aa54f1abe1d585545d2c34a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4ce48a623fa4741bed2e12aa3b7a452" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 09:55:55.349550 2707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:55:55.343203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:58273 (local address 127.0.97.2:35579)
0924 09:55:55.343403 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0924 09:55:55.343409 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 09:55:55.343947 (+ 538us) server_negotiation.cc:407] Connection header received
0924 09:55:55.344136 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:55:55.344142 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:55:55.344202 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:55:55.344277 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 09:55:55.344714 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:55:55.345404 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:55:55.347664 (+ 2260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:55:55.348316 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:55:55.348464 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:55:55.348867 (+ 403us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 09:55:55.348976 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 09:55:55.349048 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 09:55:55.349177 (+ 129us) server_negotiation.cc:299] Negotiation successful
0924 09:55:55.349260 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260924 09:55:55.349738 2718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:55:55.343212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:35579 (local address 127.0.97.1:58273)
0924 09:55:55.343856 (+ 644us) negotiation.cc:107] Waiting for socket to connect
0924 09:55:55.343871 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 09:55:55.344000 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:55:55.344305 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:55:55.344309 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:55:55.344321 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 09:55:55.344574 (+ 253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:55:55.344581 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:55:55.345532 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:55:55.345537 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:55:55.347535 (+ 1998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:55:55.347548 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:55:55.348482 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:55:55.348486 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:55:55.348627 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:55:55.348674 (+ 47us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 09:55:55.349066 (+ 392us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 09:55:55.349077 (+ 11us) client_negotiation.cc:769] Sending connection context
0924 09:55:55.349148 (+ 71us) client_negotiation.cc:240] Negotiation successful
0924 09:55:55.349297 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":329,"threads_started":1}
W20260924 09:55:55.350781 2700 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260924 09:55:56.962723 2139 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48854:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 09:55:56.962996 2139 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48854:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 09:55:56.967183 2139 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fcbcd495a6484513824d800e6faedb15: Sending DeleteTablet for 3 replicas of tablet 00191cf27d344f3d9eee0a6a5d84b5f7
I20260924 09:55:56.967947 2277 tablet_service.cc:1514] Processing DeleteTablet for tablet 00191cf27d344f3d9eee0a6a5d84b5f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 09:55:56 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:39514
I20260924 09:55:56.967973 2416 tablet_service.cc:1514] Processing DeleteTablet for tablet 00191cf27d344f3d9eee0a6a5d84b5f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 09:55:56 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:34976
I20260924 09:55:56.968138 2555 tablet_service.cc:1514] Processing DeleteTablet for tablet 00191cf27d344f3d9eee0a6a5d84b5f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 09:55:56 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:44566
I20260924 09:55:56.968739 2745 tablet_replica.cc:330] T 00191cf27d344f3d9eee0a6a5d84b5f7 P e4ce48a623fa4741bed2e12aa3b7a452: stopping tablet replica
I20260924 09:55:56.968847 2743 tablet_replica.cc:330] T 00191cf27d344f3d9eee0a6a5d84b5f7 P ccac0f0c5aa54f1abe1d585545d2c34a: stopping tablet replica
I20260924 09:55:56.968971 2745 raft_consensus.cc:2229] T 00191cf27d344f3d9eee0a6a5d84b5f7 P e4ce48a623fa4741bed2e12aa3b7a452 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 09:55:56.969048 2743 raft_consensus.cc:2229] T 00191cf27d344f3d9eee0a6a5d84b5f7 P ccac0f0c5aa54f1abe1d585545d2c34a [term 1 LEADER]: Raft consensus shutting down.
I20260924 09:55:56.969265 2745 raft_consensus.cc:2258] T 00191cf27d344f3d9eee0a6a5d84b5f7 P e4ce48a623fa4741bed2e12aa3b7a452 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 09:55:56.969579 2743 raft_consensus.cc:2258] T 00191cf27d344f3d9eee0a6a5d84b5f7 P ccac0f0c5aa54f1abe1d585545d2c34a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 09:55:56.970149 2744 tablet_replica.cc:330] T 00191cf27d344f3d9eee0a6a5d84b5f7 P 9ad27677628b497bb0f4e8636501851d: stopping tablet replica
I20260924 09:55:56.970363 2744 raft_consensus.cc:2229] T 00191cf27d344f3d9eee0a6a5d84b5f7 P 9ad27677628b497bb0f4e8636501851d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 09:55:56.970405 2745 ts_tablet_manager.cc:1905] T 00191cf27d344f3d9eee0a6a5d84b5f7 P e4ce48a623fa4741bed2e12aa3b7a452: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 09:55:56.970542 2743 ts_tablet_manager.cc:1905] T 00191cf27d344f3d9eee0a6a5d84b5f7 P ccac0f0c5aa54f1abe1d585545d2c34a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 09:55:56.970620 2744 raft_consensus.cc:2258] T 00191cf27d344f3d9eee0a6a5d84b5f7 P 9ad27677628b497bb0f4e8636501851d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 09:55:56.971499 2744 ts_tablet_manager.cc:1905] T 00191cf27d344f3d9eee0a6a5d84b5f7 P 9ad27677628b497bb0f4e8636501851d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 09:55:56.976644 2745 ts_tablet_manager.cc:1918] T 00191cf27d344f3d9eee0a6a5d84b5f7 P e4ce48a623fa4741bed2e12aa3b7a452: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 09:55:56.976688 2743 ts_tablet_manager.cc:1918] T 00191cf27d344f3d9eee0a6a5d84b5f7 P ccac0f0c5aa54f1abe1d585545d2c34a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 09:55:56.976843 2745 log.cc:1196] T 00191cf27d344f3d9eee0a6a5d84b5f7 P e4ce48a623fa4741bed2e12aa3b7a452: Deleting WAL directory at /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790243735634835-388-0/minicluster-data/ts-2/wal/wals/00191cf27d344f3d9eee0a6a5d84b5f7
I20260924 09:55:56.977273 2743 log.cc:1196] T 00191cf27d344f3d9eee0a6a5d84b5f7 P ccac0f0c5aa54f1abe1d585545d2c34a: Deleting WAL directory at /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790243735634835-388-0/minicluster-data/ts-1/wal/wals/00191cf27d344f3d9eee0a6a5d84b5f7
I20260924 09:55:56.977283 2745 ts_tablet_manager.cc:1939] T 00191cf27d344f3d9eee0a6a5d84b5f7 P e4ce48a623fa4741bed2e12aa3b7a452: Deleting consensus metadata
I20260924 09:55:56.977474 2744 ts_tablet_manager.cc:1918] T 00191cf27d344f3d9eee0a6a5d84b5f7 P 9ad27677628b497bb0f4e8636501851d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 09:55:56.977638 2744 log.cc:1196] T 00191cf27d344f3d9eee0a6a5d84b5f7 P 9ad27677628b497bb0f4e8636501851d: Deleting WAL directory at /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790243735634835-388-0/minicluster-data/ts-0/wal/wals/00191cf27d344f3d9eee0a6a5d84b5f7
I20260924 09:55:56.978088 2744 ts_tablet_manager.cc:1939] T 00191cf27d344f3d9eee0a6a5d84b5f7 P 9ad27677628b497bb0f4e8636501851d: Deleting consensus metadata
I20260924 09:55:56.978396 2125 catalog_manager.cc:4841] TS e4ce48a623fa4741bed2e12aa3b7a452 (127.0.97.3:42817): tablet 00191cf27d344f3d9eee0a6a5d84b5f7 (table test-table [id=3b2bf598588c4962a124ec1096b68a89]) successfully deleted
I20260924 09:55:56.979179 2128 catalog_manager.cc:4841] TS 9ad27677628b497bb0f4e8636501851d (127.0.97.1:44953): tablet 00191cf27d344f3d9eee0a6a5d84b5f7 (table test-table [id=3b2bf598588c4962a124ec1096b68a89]) successfully deleted
I20260924 09:55:56.979633 2743 ts_tablet_manager.cc:1939] T 00191cf27d344f3d9eee0a6a5d84b5f7 P ccac0f0c5aa54f1abe1d585545d2c34a: Deleting consensus metadata
I20260924 09:55:56.980458 2125 catalog_manager.cc:4841] TS ccac0f0c5aa54f1abe1d585545d2c34a (127.0.97.2:35579): tablet 00191cf27d344f3d9eee0a6a5d84b5f7 (table test-table [id=3b2bf598588c4962a124ec1096b68a89]) successfully deleted
Sep 24 09:55:56 dist-test-slave-w8d2 krb5kdc[2096](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243756, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 09:55:57 dist-test-slave-w8d2 krb5kdc[2096](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243756, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 09:55:57.013288 2753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:55:57.000660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48874 (local address 127.0.97.62:42045)
0924 09:55:57.001418 (+ 758us) server_negotiation.cc:206] Beginning negotiation
0924 09:55:57.001423 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 09:55:57.001442 (+ 19us) server_negotiation.cc:407] Connection header received
0924 09:55:57.001490 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:55:57.001494 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:55:57.001551 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:55:57.001638 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:55:57.002770 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:55:57.003579 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:55:57.004752 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:55:57.005126 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:55:57.005214 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:55:57.008655 (+ 3441us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:55:57.008660 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:55:57.008663 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:55:57.008693 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:55:57.011067 (+ 2374us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:55:57.011682 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:55:57.011686 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:55:57.011688 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:55:57.011799 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:55:57.012121 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:55:57.012124 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:55:57.012126 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:55:57.012375 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:55:57.012536 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0924 09:55:57.012805 (+ 269us) server_negotiation.cc:299] Negotiation successful
0924 09:55:57.012980 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":653,"thread_start_us":527,"threads_started":1}
I20260924 09:55:57.016477 2139 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48874:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 09:55:57.017083 2139 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 09:55:57.022640 2277 tablet_service.cc:1467] Processing CreateTablet for tablet b86da38b18604e11a6b40c7b24548882 (DEFAULT_TABLE table=test-table [id=f8e57b868ff74cdf85894dbcce854fbd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 09:55:57.022804 2416 tablet_service.cc:1467] Processing CreateTablet for tablet b86da38b18604e11a6b40c7b24548882 (DEFAULT_TABLE table=test-table [id=f8e57b868ff74cdf85894dbcce854fbd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 09:55:57.022864 2555 tablet_service.cc:1467] Processing CreateTablet for tablet b86da38b18604e11a6b40c7b24548882 (DEFAULT_TABLE table=test-table [id=f8e57b868ff74cdf85894dbcce854fbd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 09:55:57.023088 2277 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b86da38b18604e11a6b40c7b24548882. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 09:55:57.023201 2555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b86da38b18604e11a6b40c7b24548882. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 09:55:57.023209 2416 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b86da38b18604e11a6b40c7b24548882. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 09:55:57.029841 2755 tablet_bootstrap.cc:492] T b86da38b18604e11a6b40c7b24548882 P e4ce48a623fa4741bed2e12aa3b7a452: Bootstrap starting.
I20260924 09:55:57.030211 2754 tablet_bootstrap.cc:492] T b86da38b18604e11a6b40c7b24548882 P ccac0f0c5aa54f1abe1d585545d2c34a: Bootstrap starting.
I20260924 09:55:57.031206 2755 tablet_bootstrap.cc:654] T b86da38b18604e11a6b40c7b24548882 P e4ce48a623fa4741bed2e12aa3b7a452: Neither blocks nor log segments found. Creating new log.
I20260924 09:55:57.032354 2756 tablet_bootstrap.cc:492] T b86da38b18604e11a6b40c7b24548882 P 9ad27677628b497bb0f4e8636501851d: Bootstrap starting.
I20260924 09:55:57.032425 2754 tablet_bootstrap.cc:654] T b86da38b18604e11a6b40c7b24548882 P ccac0f0c5aa54f1abe1d585545d2c34a: Neither blocks nor log segments found. Creating new log.
I20260924 09:55:57.033895 2756 tablet_bootstrap.cc:654] T b86da38b18604e11a6b40c7b24548882 P 9ad27677628b497bb0f4e8636501851d: Neither blocks nor log segments found. Creating new log.
I20260924 09:55:57.034099 2755 tablet_bootstrap.cc:492] T b86da38b18604e11a6b40c7b24548882 P e4ce48a623fa4741bed2e12aa3b7a452: No bootstrap required, opened a new log
I20260924 09:55:57.034235 2755 ts_tablet_manager.cc:1397] T b86da38b18604e11a6b40c7b24548882 P e4ce48a623fa4741bed2e12aa3b7a452: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260924 09:55:57.034497 2754 tablet_bootstrap.cc:492] T b86da38b18604e11a6b40c7b24548882 P ccac0f0c5aa54f1abe1d585545d2c34a: No bootstrap required, opened a new log
I20260924 09:55:57.034619 2754 ts_tablet_manager.cc:1397] T b86da38b18604e11a6b40c7b24548882 P ccac0f0c5aa54f1abe1d585545d2c34a: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260924 09:55:57.034978 2755 raft_consensus.cc:348] T b86da38b18604e11a6b40c7b24548882 P e4ce48a623fa4741bed2e12aa3b7a452 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ad27677628b497bb0f4e8636501851d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44953 } } peers { permanent_uuid: "ccac0f0c5aa54f1abe1d585545d2c34a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35579 } } peers { permanent_uuid: "e4ce48a623fa4741bed2e12aa3b7a452" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42817 } }
I20260924 09:55:57.035204 2755 raft_consensus.cc:374] T b86da38b18604e11a6b40c7b24548882 P e4ce48a623fa4741bed2e12aa3b7a452 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 09:55:57.035423 2755 raft_consensus.cc:729] T b86da38b18604e11a6b40c7b24548882 P e4ce48a623fa4741bed2e12aa3b7a452 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4ce48a623fa4741bed2e12aa3b7a452, State: Initialized, Role: FOLLOWER
I20260924 09:55:57.035405 2754 raft_consensus.cc:348] T b86da38b18604e11a6b40c7b24548882 P ccac0f0c5aa54f1abe1d585545d2c34a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ad27677628b497bb0f4e8636501851d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44953 } } peers { permanent_uuid: "ccac0f0c5aa54f1abe1d585545d2c34a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35579 } } peers { permanent_uuid: "e4ce48a623fa4741bed2e12aa3b7a452" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42817 } }
I20260924 09:55:57.035607 2754 raft_consensus.cc:374] T b86da38b18604e11a6b40c7b24548882 P ccac0f0c5aa54f1abe1d585545d2c34a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 09:55:57.035638 2756 tablet_bootstrap.cc:492] T b86da38b18604e11a6b40c7b24548882 P 9ad27677628b497bb0f4e8636501851d: No bootstrap required, opened a new log
I20260924 09:55:57.035686 2754 raft_consensus.cc:729] T b86da38b18604e11a6b40c7b24548882 P ccac0f0c5aa54f1abe1d585545d2c34a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ccac0f0c5aa54f1abe1d585545d2c34a, State: Initialized, Role: FOLLOWER
I20260924 09:55:57.035749 2756 ts_tablet_manager.cc:1397] T b86da38b18604e11a6b40c7b24548882 P 9ad27677628b497bb0f4e8636501851d: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260924 09:55:57.035653 2755 consensus_queue.cc:260] T b86da38b18604e11a6b40c7b24548882 P e4ce48a623fa4741bed2e12aa3b7a452 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ad27677628b497bb0f4e8636501851d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44953 } } peers { permanent_uuid: "ccac0f0c5aa54f1abe1d585545d2c34a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35579 } } peers { permanent_uuid: "e4ce48a623fa4741bed2e12aa3b7a452" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42817 } }
I20260924 09:55:57.035888 2754 consensus_queue.cc:260] T b86da38b18604e11a6b40c7b24548882 P ccac0f0c5aa54f1abe1d585545d2c34a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ad27677628b497bb0f4e8636501851d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44953 } } peers { permanent_uuid: "ccac0f0c5aa54f1abe1d585545d2c34a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35579 } } peers { permanent_uuid: "e4ce48a623fa4741bed2e12aa3b7a452" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42817 } }
I20260924 09:55:57.036441 2756 raft_consensus.cc:348] T b86da38b18604e11a6b40c7b24548882 P 9ad27677628b497bb0f4e8636501851d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ad27677628b497bb0f4e8636501851d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44953 } } peers { permanent_uuid: "ccac0f0c5aa54f1abe1d585545d2c34a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35579 } } peers { permanent_uuid: "e4ce48a623fa4741bed2e12aa3b7a452" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42817 } }
I20260924 09:55:57.036585 2754 ts_tablet_manager.cc:1428] T b86da38b18604e11a6b40c7b24548882 P ccac0f0c5aa54f1abe1d585545d2c34a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 09:55:57.036722 2756 raft_consensus.cc:374] T b86da38b18604e11a6b40c7b24548882 P 9ad27677628b497bb0f4e8636501851d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 09:55:57.036824 2756 raft_consensus.cc:729] T b86da38b18604e11a6b40c7b24548882 P 9ad27677628b497bb0f4e8636501851d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ad27677628b497bb0f4e8636501851d, State: Initialized, Role: FOLLOWER
I20260924 09:55:57.037050 2756 consensus_queue.cc:260] T b86da38b18604e11a6b40c7b24548882 P 9ad27677628b497bb0f4e8636501851d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ad27677628b497bb0f4e8636501851d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44953 } } peers { permanent_uuid: "ccac0f0c5aa54f1abe1d585545d2c34a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35579 } } peers { permanent_uuid: "e4ce48a623fa4741bed2e12aa3b7a452" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42817 } }
I20260924 09:55:57.037612 2756 ts_tablet_manager.cc:1428] T b86da38b18604e11a6b40c7b24548882 P 9ad27677628b497bb0f4e8636501851d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 09:55:57.038993 2755 ts_tablet_manager.cc:1428] T b86da38b18604e11a6b40c7b24548882 P e4ce48a623fa4741bed2e12aa3b7a452: Time spent starting tablet: real 0.005s user 0.001s sys 0.004s
I20260924 09:55:57.075145 2760 raft_consensus.cc:482] T b86da38b18604e11a6b40c7b24548882 P e4ce48a623fa4741bed2e12aa3b7a452 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 09:55:57.075394 2760 raft_consensus.cc:504] T b86da38b18604e11a6b40c7b24548882 P e4ce48a623fa4741bed2e12aa3b7a452 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ad27677628b497bb0f4e8636501851d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44953 } } peers { permanent_uuid: "ccac0f0c5aa54f1abe1d585545d2c34a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35579 } } peers { permanent_uuid: "e4ce48a623fa4741bed2e12aa3b7a452" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42817 } }
I20260924 09:55:57.076792 2760 leader_election.cc:288] T b86da38b18604e11a6b40c7b24548882 P e4ce48a623fa4741bed2e12aa3b7a452 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9ad27677628b497bb0f4e8636501851d (127.0.97.1:44953), ccac0f0c5aa54f1abe1d585545d2c34a (127.0.97.2:35579)
I20260924 09:55:57.093803 2761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:55:57.077051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:35579 (local address 127.0.97.3:46561)
0924 09:55:57.087615 (+ 10564us) negotiation.cc:107] Waiting for socket to connect
0924 09:55:57.087639 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0924 09:55:57.087793 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:55:57.088186 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:55:57.088191 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:55:57.088212 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 09:55:57.088429 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:55:57.088437 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:55:57.089713 (+ 1276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:55:57.089717 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:55:57.091798 (+ 2081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:55:57.091812 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:55:57.093281 (+ 1469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:55:57.093286 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:55:57.093397 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:55:57.093422 (+ 25us) client_negotiation.cc:769] Sending connection context
0924 09:55:57.093491 (+ 69us) client_negotiation.cc:240] Negotiation successful
0924 09:55:57.093578 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10411,"thread_start_us":10267,"threads_started":1}
I20260924 09:55:57.094125 2764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:55:57.077210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:46561 (local address 127.0.97.2:35579)
0924 09:55:57.087362 (+ 10152us) server_negotiation.cc:206] Beginning negotiation
0924 09:55:57.087369 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 09:55:57.087929 (+ 560us) server_negotiation.cc:407] Connection header received
0924 09:55:57.088007 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:55:57.088012 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:55:57.088081 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:55:57.088172 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 09:55:57.088584 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:55:57.089584 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:55:57.091935 (+ 2351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:55:57.093120 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:55:57.093246 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:55:57.093379 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0924 09:55:57.093525 (+ 146us) server_negotiation.cc:299] Negotiation successful
0924 09:55:57.093653 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9993,"thread_start_us":9858,"threads_started":1}
I20260924 09:55:57.094601 2763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:55:57.077084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44953 (local address 127.0.97.3:57433)
0924 09:55:57.088008 (+ 10924us) negotiation.cc:107] Waiting for socket to connect
0924 09:55:57.088025 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 09:55:57.088137 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:55:57.088520 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:55:57.088524 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:55:57.088546 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 09:55:57.088836 (+ 290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:55:57.088844 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:55:57.090041 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:55:57.090049 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:55:57.091963 (+ 1914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:55:57.091974 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:55:57.093930 (+ 1956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:55:57.093937 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:55:57.094115 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:55:57.094139 (+ 24us) client_negotiation.cc:769] Sending connection context
0924 09:55:57.094219 (+ 80us) client_negotiation.cc:240] Negotiation successful
0924 09:55:57.094331 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10789,"thread_start_us":10581,"threads_started":1}
I20260924 09:55:57.094661 2762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:55:57.077087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:57433 (local address 127.0.97.1:44953)
0924 09:55:57.088240 (+ 11153us) server_negotiation.cc:206] Beginning negotiation
0924 09:55:57.088246 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 09:55:57.088266 (+ 20us) server_negotiation.cc:407] Connection header received
0924 09:55:57.088330 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:55:57.088335 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:55:57.088407 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:55:57.088479 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 09:55:57.088978 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:55:57.089920 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:55:57.092082 (+ 2162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:55:57.093736 (+ 1654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:55:57.093849 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:55:57.093941 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0924 09:55:57.094246 (+ 305us) server_negotiation.cc:299] Negotiation successful
0924 09:55:57.094377 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11029,"thread_start_us":10896,"threads_started":1}
I20260924 09:55:57.094676 2435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b86da38b18604e11a6b40c7b24548882" candidate_uuid: "e4ce48a623fa4741bed2e12aa3b7a452" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ccac0f0c5aa54f1abe1d585545d2c34a" is_pre_election: true
I20260924 09:55:57.094920 2435 raft_consensus.cc:2454] T b86da38b18604e11a6b40c7b24548882 P ccac0f0c5aa54f1abe1d585545d2c34a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4ce48a623fa4741bed2e12aa3b7a452 in term 0.
I20260924 09:55:57.095163 2296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b86da38b18604e11a6b40c7b24548882" candidate_uuid: "e4ce48a623fa4741bed2e12aa3b7a452" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ad27677628b497bb0f4e8636501851d" is_pre_election: true
I20260924 09:55:57.095412 2296 raft_consensus.cc:2454] T b86da38b18604e11a6b40c7b24548882 P 9ad27677628b497bb0f4e8636501851d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4ce48a623fa4741bed2e12aa3b7a452 in term 0.
I20260924 09:55:57.095438 2506 leader_election.cc:302] T b86da38b18604e11a6b40c7b24548882 P e4ce48a623fa4741bed2e12aa3b7a452 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ccac0f0c5aa54f1abe1d585545d2c34a, e4ce48a623fa4741bed2e12aa3b7a452; no voters:
I20260924 09:55:57.095815 2760 raft_consensus.cc:2781] T b86da38b18604e11a6b40c7b24548882 P e4ce48a623fa4741bed2e12aa3b7a452 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 09:55:57.095920 2760 raft_consensus.cc:482] T b86da38b18604e11a6b40c7b24548882 P e4ce48a623fa4741bed2e12aa3b7a452 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 09:55:57.096011 2760 raft_consensus.cc:3037] T b86da38b18604e11a6b40c7b24548882 P e4ce48a623fa4741bed2e12aa3b7a452 [term 0 FOLLOWER]: Advancing to term 1
I20260924 09:55:57.097970 2760 raft_consensus.cc:504] T b86da38b18604e11a6b40c7b24548882 P e4ce48a623fa4741bed2e12aa3b7a452 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ad27677628b497bb0f4e8636501851d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44953 } } peers { permanent_uuid: "ccac0f0c5aa54f1abe1d585545d2c34a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35579 } } peers { permanent_uuid: "e4ce48a623fa4741bed2e12aa3b7a452" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42817 } }
I20260924 09:55:57.098531 2760 leader_election.cc:288] T b86da38b18604e11a6b40c7b24548882 P e4ce48a623fa4741bed2e12aa3b7a452 [CANDIDATE]: Term 1 election: Requested vote from peers 9ad27677628b497bb0f4e8636501851d (127.0.97.1:44953), ccac0f0c5aa54f1abe1d585545d2c34a (127.0.97.2:35579)
I20260924 09:55:57.098920 2296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b86da38b18604e11a6b40c7b24548882" candidate_uuid: "e4ce48a623fa4741bed2e12aa3b7a452" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ad27677628b497bb0f4e8636501851d"
I20260924 09:55:57.099097 2296 raft_consensus.cc:3037] T b86da38b18604e11a6b40c7b24548882 P 9ad27677628b497bb0f4e8636501851d [term 0 FOLLOWER]: Advancing to term 1
I20260924 09:55:57.099027 2435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b86da38b18604e11a6b40c7b24548882" candidate_uuid: "e4ce48a623fa4741bed2e12aa3b7a452" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ccac0f0c5aa54f1abe1d585545d2c34a"
I20260924 09:55:57.099221 2435 raft_consensus.cc:3037] T b86da38b18604e11a6b40c7b24548882 P ccac0f0c5aa54f1abe1d585545d2c34a [term 0 FOLLOWER]: Advancing to term 1
I20260924 09:55:57.101020 2296 raft_consensus.cc:2454] T b86da38b18604e11a6b40c7b24548882 P 9ad27677628b497bb0f4e8636501851d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4ce48a623fa4741bed2e12aa3b7a452 in term 1.
I20260924 09:55:57.101136 2435 raft_consensus.cc:2454] T b86da38b18604e11a6b40c7b24548882 P ccac0f0c5aa54f1abe1d585545d2c34a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4ce48a623fa4741bed2e12aa3b7a452 in term 1.
I20260924 09:55:57.101361 2509 leader_election.cc:302] T b86da38b18604e11a6b40c7b24548882 P e4ce48a623fa4741bed2e12aa3b7a452 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ad27677628b497bb0f4e8636501851d, e4ce48a623fa4741bed2e12aa3b7a452; no voters:
I20260924 09:55:57.101655 2760 raft_consensus.cc:2781] T b86da38b18604e11a6b40c7b24548882 P e4ce48a623fa4741bed2e12aa3b7a452 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 09:55:57.102118 2760 raft_consensus.cc:686] T b86da38b18604e11a6b40c7b24548882 P e4ce48a623fa4741bed2e12aa3b7a452 [term 1 LEADER]: Becoming Leader. State: Replica: e4ce48a623fa4741bed2e12aa3b7a452, State: Running, Role: LEADER
I20260924 09:55:57.102540 2760 consensus_queue.cc:237] T b86da38b18604e11a6b40c7b24548882 P e4ce48a623fa4741bed2e12aa3b7a452 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ad27677628b497bb0f4e8636501851d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44953 } } peers { permanent_uuid: "ccac0f0c5aa54f1abe1d585545d2c34a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35579 } } peers { permanent_uuid: "e4ce48a623fa4741bed2e12aa3b7a452" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42817 } }
I20260924 09:55:57.106117 2141 catalog_manager.cc:5495] T b86da38b18604e11a6b40c7b24548882 P e4ce48a623fa4741bed2e12aa3b7a452 reported cstate change: term changed from 0 to 1, leader changed from <none> to e4ce48a623fa4741bed2e12aa3b7a452 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "e4ce48a623fa4741bed2e12aa3b7a452" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ad27677628b497bb0f4e8636501851d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44953 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ccac0f0c5aa54f1abe1d585545d2c34a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35579 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4ce48a623fa4741bed2e12aa3b7a452" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42817 } health_report { overall_health: HEALTHY } } }
I20260924 09:55:57.137677 2768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:55:57.132496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44572 (local address 127.0.97.3:42817)
0924 09:55:57.132949 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0924 09:55:57.132953 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 09:55:57.133018 (+ 65us) server_negotiation.cc:407] Connection header received
0924 09:55:57.133100 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:55:57.133103 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:55:57.133171 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:55:57.133299 (+ 128us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 09:55:57.133608 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:55:57.134444 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:55:57.135666 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:55:57.136058 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:55:57.136218 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:55:57.136543 (+ 325us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 09:55:57.137103 (+ 560us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 09:55:57.137224 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0924 09:55:57.137317 (+ 93us) server_negotiation.cc:299] Negotiation successful
0924 09:55:57.137374 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":362,"thread_start_us":494,"threads_started":1}
I20260924 09:55:57.150972 2762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:55:57.146096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:35231 (local address 127.0.97.1:44953)
0924 09:55:57.146280 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0924 09:55:57.146287 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 09:55:57.146620 (+ 333us) server_negotiation.cc:407] Connection header received
0924 09:55:57.146775 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:55:57.146778 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:55:57.146834 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:55:57.146908 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 09:55:57.147332 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:55:57.148390 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:55:57.149625 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:55:57.150010 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:55:57.150098 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:55:57.150496 (+ 398us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 09:55:57.150585 (+ 89us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 09:55:57.150642 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 09:55:57.150726 (+ 84us) server_negotiation.cc:299] Negotiation successful
0924 09:55:57.150783 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 09:55:57.151082 2771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:55:57.145947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44953 (local address 127.0.97.3:35231)
0924 09:55:57.146530 (+ 583us) negotiation.cc:107] Waiting for socket to connect
0924 09:55:57.146547 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 09:55:57.146665 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:55:57.146930 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:55:57.146934 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:55:57.146944 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 09:55:57.147182 (+ 238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:55:57.147189 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:55:57.148512 (+ 1323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:55:57.148516 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:55:57.149513 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:55:57.149523 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:55:57.150179 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:55:57.150184 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:55:57.150305 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:55:57.150361 (+ 56us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 09:55:57.150649 (+ 288us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 09:55:57.150656 (+ 7us) client_negotiation.cc:769] Sending connection context
0924 09:55:57.150707 (+ 51us) client_negotiation.cc:240] Negotiation successful
0924 09:55:57.150806 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":309,"threads_started":1}
I20260924 09:55:57.158334 2435 raft_consensus.cc:1261] T b86da38b18604e11a6b40c7b24548882 P ccac0f0c5aa54f1abe1d585545d2c34a [term 1 FOLLOWER]: Refusing update from remote peer e4ce48a623fa4741bed2e12aa3b7a452: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 09:55:57.158327 2296 raft_consensus.cc:1261] T b86da38b18604e11a6b40c7b24548882 P 9ad27677628b497bb0f4e8636501851d [term 1 FOLLOWER]: Refusing update from remote peer e4ce48a623fa4741bed2e12aa3b7a452: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 09:55:57.158906 2765 consensus_queue.cc:1035] T b86da38b18604e11a6b40c7b24548882 P e4ce48a623fa4741bed2e12aa3b7a452 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ccac0f0c5aa54f1abe1d585545d2c34a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 09:55:57.159307 2760 consensus_queue.cc:1035] T b86da38b18604e11a6b40c7b24548882 P e4ce48a623fa4741bed2e12aa3b7a452 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ad27677628b497bb0f4e8636501851d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 09:55:57.183153 2768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:55:57.177896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:34395 (local address 127.0.97.3:42817)
0924 09:55:57.178052 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0924 09:55:57.178058 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 09:55:57.178399 (+ 341us) server_negotiation.cc:407] Connection header received
0924 09:55:57.178567 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:55:57.178572 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:55:57.178633 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:55:57.178698 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 09:55:57.179158 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:55:57.180028 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:55:57.181668 (+ 1640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:55:57.182064 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:55:57.182164 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:55:57.182510 (+ 346us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 09:55:57.182603 (+ 93us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 09:55:57.182669 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 09:55:57.182796 (+ 127us) server_negotiation.cc:299] Negotiation successful
0924 09:55:57.182866 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260924 09:55:57.183216 2776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:55:57.177816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:42817 (local address 127.0.97.1:34395)
0924 09:55:57.178318 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0924 09:55:57.178332 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 09:55:57.178446 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:55:57.178720 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:55:57.178724 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:55:57.178735 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 09:55:57.179003 (+ 268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:55:57.179012 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:55:57.180148 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:55:57.180153 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:55:57.181552 (+ 1399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:55:57.181563 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:55:57.182214 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:55:57.182221 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:55:57.182343 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:55:57.182376 (+ 33us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 09:55:57.182682 (+ 306us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 09:55:57.182693 (+ 11us) client_negotiation.cc:769] Sending connection context
0924 09:55:57.182772 (+ 79us) client_negotiation.cc:240] Negotiation successful
0924 09:55:57.182893 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":270,"threads_started":1}
I20260924 09:55:57.212306 2141 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48874:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 09:55:57.212613 2141 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48874:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 09:55:57.216089 2141 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fcbcd495a6484513824d800e6faedb15: Sending DeleteTablet for 3 replicas of tablet b86da38b18604e11a6b40c7b24548882
I20260924 09:55:57.216821 2277 tablet_service.cc:1514] Processing DeleteTablet for tablet b86da38b18604e11a6b40c7b24548882 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 09:55:57 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:39514
I20260924 09:55:57.217046 2416 tablet_service.cc:1514] Processing DeleteTablet for tablet b86da38b18604e11a6b40c7b24548882 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 09:55:57 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:34976
I20260924 09:55:57.217115 2744 tablet_replica.cc:330] T b86da38b18604e11a6b40c7b24548882 P 9ad27677628b497bb0f4e8636501851d: stopping tablet replica
I20260924 09:55:57.217293 2744 raft_consensus.cc:2229] T b86da38b18604e11a6b40c7b24548882 P 9ad27677628b497bb0f4e8636501851d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 09:55:57.217396 2743 tablet_replica.cc:330] T b86da38b18604e11a6b40c7b24548882 P ccac0f0c5aa54f1abe1d585545d2c34a: stopping tablet replica
I20260924 09:55:57.217466 2744 raft_consensus.cc:2258] T b86da38b18604e11a6b40c7b24548882 P 9ad27677628b497bb0f4e8636501851d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 09:55:57.217592 2743 raft_consensus.cc:2229] T b86da38b18604e11a6b40c7b24548882 P ccac0f0c5aa54f1abe1d585545d2c34a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 09:55:57.217760 2743 raft_consensus.cc:2258] T b86da38b18604e11a6b40c7b24548882 P ccac0f0c5aa54f1abe1d585545d2c34a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 09:55:57.218417 2555 tablet_service.cc:1514] Processing DeleteTablet for tablet b86da38b18604e11a6b40c7b24548882 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 09:55:57 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:44566
I20260924 09:55:57.218288 2141 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48874:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260924 09:55:57.218596 2744 ts_tablet_manager.cc:1905] T b86da38b18604e11a6b40c7b24548882 P 9ad27677628b497bb0f4e8636501851d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 09:55:57.218710 2745 tablet_replica.cc:330] T b86da38b18604e11a6b40c7b24548882 P e4ce48a623fa4741bed2e12aa3b7a452: stopping tablet replica
I20260924 09:55:57.218892 2745 raft_consensus.cc:2229] T b86da38b18604e11a6b40c7b24548882 P e4ce48a623fa4741bed2e12aa3b7a452 [term 1 LEADER]: Raft consensus shutting down.
W20260924 09:55:57.218906 2141 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 09:55:57.219403 2745 raft_consensus.cc:2258] T b86da38b18604e11a6b40c7b24548882 P e4ce48a623fa4741bed2e12aa3b7a452 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 09:55:57.220515 2743 ts_tablet_manager.cc:1905] T b86da38b18604e11a6b40c7b24548882 P ccac0f0c5aa54f1abe1d585545d2c34a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 09:55:57.223466 2744 ts_tablet_manager.cc:1918] T b86da38b18604e11a6b40c7b24548882 P 9ad27677628b497bb0f4e8636501851d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 09:55:57.223906 2744 log.cc:1196] T b86da38b18604e11a6b40c7b24548882 P 9ad27677628b497bb0f4e8636501851d: Deleting WAL directory at /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790243735634835-388-0/minicluster-data/ts-0/wal/wals/b86da38b18604e11a6b40c7b24548882
I20260924 09:55:57.224258 2744 ts_tablet_manager.cc:1939] T b86da38b18604e11a6b40c7b24548882 P 9ad27677628b497bb0f4e8636501851d: Deleting consensus metadata
I20260924 09:55:57.225179 2745 ts_tablet_manager.cc:1905] T b86da38b18604e11a6b40c7b24548882 P e4ce48a623fa4741bed2e12aa3b7a452: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 09:55:57.225193 2128 catalog_manager.cc:4841] TS 9ad27677628b497bb0f4e8636501851d (127.0.97.1:44953): tablet b86da38b18604e11a6b40c7b24548882 (table test-table [id=f8e57b868ff74cdf85894dbcce854fbd]) successfully deleted
I20260924 09:55:57.229493 2743 ts_tablet_manager.cc:1918] T b86da38b18604e11a6b40c7b24548882 P ccac0f0c5aa54f1abe1d585545d2c34a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 09:55:57.229655 2743 log.cc:1196] T b86da38b18604e11a6b40c7b24548882 P ccac0f0c5aa54f1abe1d585545d2c34a: Deleting WAL directory at /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790243735634835-388-0/minicluster-data/ts-1/wal/wals/b86da38b18604e11a6b40c7b24548882
I20260924 09:55:57.230065 2743 ts_tablet_manager.cc:1939] T b86da38b18604e11a6b40c7b24548882 P ccac0f0c5aa54f1abe1d585545d2c34a: Deleting consensus metadata
I20260924 09:55:57.232100 2125 catalog_manager.cc:4841] TS ccac0f0c5aa54f1abe1d585545d2c34a (127.0.97.2:35579): tablet b86da38b18604e11a6b40c7b24548882 (table test-table [id=f8e57b868ff74cdf85894dbcce854fbd]) successfully deleted
I20260924 09:55:57.232613 2745 ts_tablet_manager.cc:1918] T b86da38b18604e11a6b40c7b24548882 P e4ce48a623fa4741bed2e12aa3b7a452: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 09:55:57.232867 2745 log.cc:1196] T b86da38b18604e11a6b40c7b24548882 P e4ce48a623fa4741bed2e12aa3b7a452: Deleting WAL directory at /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790243735634835-388-0/minicluster-data/ts-2/wal/wals/b86da38b18604e11a6b40c7b24548882
I20260924 09:55:57.233321 2745 ts_tablet_manager.cc:1939] T b86da38b18604e11a6b40c7b24548882 P e4ce48a623fa4741bed2e12aa3b7a452: Deleting consensus metadata
I20260924 09:55:57.234243 2277 tablet_service.cc:1467] Processing CreateTablet for tablet 89ce407c649b4b34a034336e328461c3 (DEFAULT_TABLE table=test-table [id=a25eb3ea885445168d29b622061b162b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 09:55:57.234656 2277 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89ce407c649b4b34a034336e328461c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 09:55:57.234978 2125 catalog_manager.cc:4841] TS e4ce48a623fa4741bed2e12aa3b7a452 (127.0.97.3:42817): tablet b86da38b18604e11a6b40c7b24548882 (table test-table [id=f8e57b868ff74cdf85894dbcce854fbd]) successfully deleted
I20260924 09:55:57.236210 2555 tablet_service.cc:1467] Processing CreateTablet for tablet 89ce407c649b4b34a034336e328461c3 (DEFAULT_TABLE table=test-table [id=a25eb3ea885445168d29b622061b162b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 09:55:57.236577 2555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89ce407c649b4b34a034336e328461c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 09:55:57.239928 2416 tablet_service.cc:1467] Processing CreateTablet for tablet 89ce407c649b4b34a034336e328461c3 (DEFAULT_TABLE table=test-table [id=a25eb3ea885445168d29b622061b162b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 09:55:57.240283 2416 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89ce407c649b4b34a034336e328461c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 09:55:57.243317 2756 tablet_bootstrap.cc:492] T 89ce407c649b4b34a034336e328461c3 P 9ad27677628b497bb0f4e8636501851d: Bootstrap starting.
I20260924 09:55:57.243713 2755 tablet_bootstrap.cc:492] T 89ce407c649b4b34a034336e328461c3 P e4ce48a623fa4741bed2e12aa3b7a452: Bootstrap starting.
I20260924 09:55:57.244760 2756 tablet_bootstrap.cc:654] T 89ce407c649b4b34a034336e328461c3 P 9ad27677628b497bb0f4e8636501851d: Neither blocks nor log segments found. Creating new log.
I20260924 09:55:57.245210 2755 tablet_bootstrap.cc:654] T 89ce407c649b4b34a034336e328461c3 P e4ce48a623fa4741bed2e12aa3b7a452: Neither blocks nor log segments found. Creating new log.
I20260924 09:55:57.249154 2754 tablet_bootstrap.cc:492] T 89ce407c649b4b34a034336e328461c3 P ccac0f0c5aa54f1abe1d585545d2c34a: Bootstrap starting.
I20260924 09:55:57.250780 2754 tablet_bootstrap.cc:654] T 89ce407c649b4b34a034336e328461c3 P ccac0f0c5aa54f1abe1d585545d2c34a: Neither blocks nor log segments found. Creating new log.
I20260924 09:55:57.257259 2756 tablet_bootstrap.cc:492] T 89ce407c649b4b34a034336e328461c3 P 9ad27677628b497bb0f4e8636501851d: No bootstrap required, opened a new log
I20260924 09:55:57.257434 2756 ts_tablet_manager.cc:1397] T 89ce407c649b4b34a034336e328461c3 P 9ad27677628b497bb0f4e8636501851d: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.004s
I20260924 09:55:57.258219 2756 raft_consensus.cc:348] T 89ce407c649b4b34a034336e328461c3 P 9ad27677628b497bb0f4e8636501851d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ad27677628b497bb0f4e8636501851d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44953 } } peers { permanent_uuid: "ccac0f0c5aa54f1abe1d585545d2c34a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35579 } } peers { permanent_uuid: "e4ce48a623fa4741bed2e12aa3b7a452" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42817 } }
I20260924 09:55:57.258548 2756 raft_consensus.cc:374] T 89ce407c649b4b34a034336e328461c3 P 9ad27677628b497bb0f4e8636501851d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 09:55:57.258616 2756 raft_consensus.cc:729] T 89ce407c649b4b34a034336e328461c3 P 9ad27677628b497bb0f4e8636501851d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ad27677628b497bb0f4e8636501851d, State: Initialized, Role: FOLLOWER
I20260924 09:55:57.258885 2756 consensus_queue.cc:260] T 89ce407c649b4b34a034336e328461c3 P 9ad27677628b497bb0f4e8636501851d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ad27677628b497bb0f4e8636501851d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44953 } } peers { permanent_uuid: "ccac0f0c5aa54f1abe1d585545d2c34a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35579 } } peers { permanent_uuid: "e4ce48a623fa4741bed2e12aa3b7a452" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42817 } }
I20260924 09:55:57.259637 2756 ts_tablet_manager.cc:1428] T 89ce407c649b4b34a034336e328461c3 P 9ad27677628b497bb0f4e8636501851d: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260924 09:55:57.261550 2755 tablet_bootstrap.cc:492] T 89ce407c649b4b34a034336e328461c3 P e4ce48a623fa4741bed2e12aa3b7a452: No bootstrap required, opened a new log
I20260924 09:55:57.261726 2755 ts_tablet_manager.cc:1397] T 89ce407c649b4b34a034336e328461c3 P e4ce48a623fa4741bed2e12aa3b7a452: Time spent bootstrapping tablet: real 0.018s user 0.005s sys 0.005s
I20260924 09:55:57.262503 2755 raft_consensus.cc:348] T 89ce407c649b4b34a034336e328461c3 P e4ce48a623fa4741bed2e12aa3b7a452 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ad27677628b497bb0f4e8636501851d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44953 } } peers { permanent_uuid: "ccac0f0c5aa54f1abe1d585545d2c34a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35579 } } peers { permanent_uuid: "e4ce48a623fa4741bed2e12aa3b7a452" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42817 } }
I20260924 09:55:57.262814 2755 raft_consensus.cc:374] T 89ce407c649b4b34a034336e328461c3 P e4ce48a623fa4741bed2e12aa3b7a452 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 09:55:57.262902 2755 raft_consensus.cc:729] T 89ce407c649b4b34a034336e328461c3 P e4ce48a623fa4741bed2e12aa3b7a452 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4ce48a623fa4741bed2e12aa3b7a452, State: Initialized, Role: FOLLOWER
I20260924 09:55:57.263167 2755 consensus_queue.cc:260] T 89ce407c649b4b34a034336e328461c3 P e4ce48a623fa4741bed2e12aa3b7a452 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ad27677628b497bb0f4e8636501851d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44953 } } peers { permanent_uuid: "ccac0f0c5aa54f1abe1d585545d2c34a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35579 } } peers { permanent_uuid: "e4ce48a623fa4741bed2e12aa3b7a452" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42817 } }
I20260924 09:55:57.263782 2755 ts_tablet_manager.cc:1428] T 89ce407c649b4b34a034336e328461c3 P e4ce48a623fa4741bed2e12aa3b7a452: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 09:55:57.280432 2754 tablet_bootstrap.cc:492] T 89ce407c649b4b34a034336e328461c3 P ccac0f0c5aa54f1abe1d585545d2c34a: No bootstrap required, opened a new log
I20260924 09:55:57.280630 2754 ts_tablet_manager.cc:1397] T 89ce407c649b4b34a034336e328461c3 P ccac0f0c5aa54f1abe1d585545d2c34a: Time spent bootstrapping tablet: real 0.032s user 0.003s sys 0.006s
I20260924 09:55:57.281415 2754 raft_consensus.cc:348] T 89ce407c649b4b34a034336e328461c3 P ccac0f0c5aa54f1abe1d585545d2c34a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ad27677628b497bb0f4e8636501851d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44953 } } peers { permanent_uuid: "ccac0f0c5aa54f1abe1d585545d2c34a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35579 } } peers { permanent_uuid: "e4ce48a623fa4741bed2e12aa3b7a452" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42817 } }
I20260924 09:55:57.281664 2754 raft_consensus.cc:374] T 89ce407c649b4b34a034336e328461c3 P ccac0f0c5aa54f1abe1d585545d2c34a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 09:55:57.281734 2754 raft_consensus.cc:729] T 89ce407c649b4b34a034336e328461c3 P ccac0f0c5aa54f1abe1d585545d2c34a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ccac0f0c5aa54f1abe1d585545d2c34a, State: Initialized, Role: FOLLOWER
I20260924 09:55:57.281971 2754 consensus_queue.cc:260] T 89ce407c649b4b34a034336e328461c3 P ccac0f0c5aa54f1abe1d585545d2c34a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ad27677628b497bb0f4e8636501851d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44953 } } peers { permanent_uuid: "ccac0f0c5aa54f1abe1d585545d2c34a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35579 } } peers { permanent_uuid: "e4ce48a623fa4741bed2e12aa3b7a452" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42817 } }
I20260924 09:55:57.282511 2754 ts_tablet_manager.cc:1428] T 89ce407c649b4b34a034336e328461c3 P ccac0f0c5aa54f1abe1d585545d2c34a: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260924 09:55:57.332548 2729 raft_consensus.cc:482] T 89ce407c649b4b34a034336e328461c3 P 9ad27677628b497bb0f4e8636501851d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 09:55:57.332748 2729 raft_consensus.cc:504] T 89ce407c649b4b34a034336e328461c3 P 9ad27677628b497bb0f4e8636501851d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ad27677628b497bb0f4e8636501851d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44953 } } peers { permanent_uuid: "ccac0f0c5aa54f1abe1d585545d2c34a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35579 } } peers { permanent_uuid: "e4ce48a623fa4741bed2e12aa3b7a452" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42817 } }
I20260924 09:55:57.333461 2729 leader_election.cc:288] T 89ce407c649b4b34a034336e328461c3 P 9ad27677628b497bb0f4e8636501851d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ccac0f0c5aa54f1abe1d585545d2c34a (127.0.97.2:35579), e4ce48a623fa4741bed2e12aa3b7a452 (127.0.97.3:42817)
I20260924 09:55:57.334017 2435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89ce407c649b4b34a034336e328461c3" candidate_uuid: "9ad27677628b497bb0f4e8636501851d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ccac0f0c5aa54f1abe1d585545d2c34a" is_pre_election: true
I20260924 09:55:57.334102 2574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89ce407c649b4b34a034336e328461c3" candidate_uuid: "9ad27677628b497bb0f4e8636501851d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4ce48a623fa4741bed2e12aa3b7a452" is_pre_election: true
I20260924 09:55:57.334224 2435 raft_consensus.cc:2454] T 89ce407c649b4b34a034336e328461c3 P ccac0f0c5aa54f1abe1d585545d2c34a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ad27677628b497bb0f4e8636501851d in term 0.
I20260924 09:55:57.334358 2574 raft_consensus.cc:2454] T 89ce407c649b4b34a034336e328461c3 P e4ce48a623fa4741bed2e12aa3b7a452 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ad27677628b497bb0f4e8636501851d in term 0.
I20260924 09:55:57.334693 2228 leader_election.cc:302] T 89ce407c649b4b34a034336e328461c3 P 9ad27677628b497bb0f4e8636501851d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ad27677628b497bb0f4e8636501851d, ccac0f0c5aa54f1abe1d585545d2c34a; no voters:
I20260924 09:55:57.334941 2729 raft_consensus.cc:2781] T 89ce407c649b4b34a034336e328461c3 P 9ad27677628b497bb0f4e8636501851d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 09:55:57.335094 2729 raft_consensus.cc:482] T 89ce407c649b4b34a034336e328461c3 P 9ad27677628b497bb0f4e8636501851d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 09:55:57.335186 2729 raft_consensus.cc:3037] T 89ce407c649b4b34a034336e328461c3 P 9ad27677628b497bb0f4e8636501851d [term 0 FOLLOWER]: Advancing to term 1
I20260924 09:55:57.337231 2729 raft_consensus.cc:504] T 89ce407c649b4b34a034336e328461c3 P 9ad27677628b497bb0f4e8636501851d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ad27677628b497bb0f4e8636501851d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44953 } } peers { permanent_uuid: "ccac0f0c5aa54f1abe1d585545d2c34a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35579 } } peers { permanent_uuid: "e4ce48a623fa4741bed2e12aa3b7a452" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42817 } }
I20260924 09:55:57.337714 2729 leader_election.cc:288] T 89ce407c649b4b34a034336e328461c3 P 9ad27677628b497bb0f4e8636501851d [CANDIDATE]: Term 1 election: Requested vote from peers ccac0f0c5aa54f1abe1d585545d2c34a (127.0.97.2:35579), e4ce48a623fa4741bed2e12aa3b7a452 (127.0.97.3:42817)
I20260924 09:55:57.338124 2435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89ce407c649b4b34a034336e328461c3" candidate_uuid: "9ad27677628b497bb0f4e8636501851d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ccac0f0c5aa54f1abe1d585545d2c34a"
I20260924 09:55:57.338214 2574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89ce407c649b4b34a034336e328461c3" candidate_uuid: "9ad27677628b497bb0f4e8636501851d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4ce48a623fa4741bed2e12aa3b7a452"
I20260924 09:55:57.338290 2435 raft_consensus.cc:3037] T 89ce407c649b4b34a034336e328461c3 P ccac0f0c5aa54f1abe1d585545d2c34a [term 0 FOLLOWER]: Advancing to term 1
I20260924 09:55:57.338413 2574 raft_consensus.cc:3037] T 89ce407c649b4b34a034336e328461c3 P e4ce48a623fa4741bed2e12aa3b7a452 [term 0 FOLLOWER]: Advancing to term 1
I20260924 09:55:57.340678 2435 raft_consensus.cc:2454] T 89ce407c649b4b34a034336e328461c3 P ccac0f0c5aa54f1abe1d585545d2c34a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ad27677628b497bb0f4e8636501851d in term 1.
I20260924 09:55:57.341064 2228 leader_election.cc:302] T 89ce407c649b4b34a034336e328461c3 P 9ad27677628b497bb0f4e8636501851d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ad27677628b497bb0f4e8636501851d, ccac0f0c5aa54f1abe1d585545d2c34a; no voters:
I20260924 09:55:57.341317 2729 raft_consensus.cc:2781] T 89ce407c649b4b34a034336e328461c3 P 9ad27677628b497bb0f4e8636501851d [term 1 FOLLOWER]: Leader election won for term 1
I20260924 09:55:57.341459 2729 raft_consensus.cc:686] T 89ce407c649b4b34a034336e328461c3 P 9ad27677628b497bb0f4e8636501851d [term 1 LEADER]: Becoming Leader. State: Replica: 9ad27677628b497bb0f4e8636501851d, State: Running, Role: LEADER
I20260924 09:55:57.341711 2729 consensus_queue.cc:237] T 89ce407c649b4b34a034336e328461c3 P 9ad27677628b497bb0f4e8636501851d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ad27677628b497bb0f4e8636501851d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44953 } } peers { permanent_uuid: "ccac0f0c5aa54f1abe1d585545d2c34a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35579 } } peers { permanent_uuid: "e4ce48a623fa4741bed2e12aa3b7a452" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42817 } }
I20260924 09:55:57.341966 2574 raft_consensus.cc:2454] T 89ce407c649b4b34a034336e328461c3 P e4ce48a623fa4741bed2e12aa3b7a452 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ad27677628b497bb0f4e8636501851d in term 1.
I20260924 09:55:57.344190 2139 catalog_manager.cc:5495] T 89ce407c649b4b34a034336e328461c3 P 9ad27677628b497bb0f4e8636501851d reported cstate change: term changed from 0 to 1, leader changed from <none> to 9ad27677628b497bb0f4e8636501851d (127.0.97.1). New cstate: current_term: 1 leader_uuid: "9ad27677628b497bb0f4e8636501851d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ad27677628b497bb0f4e8636501851d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44953 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ccac0f0c5aa54f1abe1d585545d2c34a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35579 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4ce48a623fa4741bed2e12aa3b7a452" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42817 } health_report { overall_health: UNKNOWN } } }
I20260924 09:55:57.381796 2762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:55:57.376727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39538 (local address 127.0.97.1:44953)
0924 09:55:57.376886 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0924 09:55:57.376892 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 09:55:57.377008 (+ 116us) server_negotiation.cc:407] Connection header received
0924 09:55:57.377166 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:55:57.377171 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:55:57.377250 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:55:57.377335 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 09:55:57.377625 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:55:57.378732 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:55:57.380018 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:55:57.380645 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:55:57.380771 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:55:57.381117 (+ 346us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 09:55:57.381243 (+ 126us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 09:55:57.381327 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0924 09:55:57.381421 (+ 94us) server_negotiation.cc:299] Negotiation successful
0924 09:55:57.381506 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 09:55:57.390189 2574 raft_consensus.cc:1261] T 89ce407c649b4b34a034336e328461c3 P e4ce48a623fa4741bed2e12aa3b7a452 [term 1 FOLLOWER]: Refusing update from remote peer 9ad27677628b497bb0f4e8636501851d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 09:55:57.390369 2435 raft_consensus.cc:1261] T 89ce407c649b4b34a034336e328461c3 P ccac0f0c5aa54f1abe1d585545d2c34a [term 1 FOLLOWER]: Refusing update from remote peer 9ad27677628b497bb0f4e8636501851d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 09:55:57.390782 2689 consensus_queue.cc:1035] T 89ce407c649b4b34a034336e328461c3 P 9ad27677628b497bb0f4e8636501851d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4ce48a623fa4741bed2e12aa3b7a452" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 09:55:57.391317 2724 consensus_queue.cc:1035] T 89ce407c649b4b34a034336e328461c3 P 9ad27677628b497bb0f4e8636501851d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ccac0f0c5aa54f1abe1d585545d2c34a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 09:55:57.411173 2722 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260924 09:55:58.985957 2139 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48874:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 09:55:58.986169 2139 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48874:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 09:55:58.989843 2139 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fcbcd495a6484513824d800e6faedb15: Sending DeleteTablet for 3 replicas of tablet 89ce407c649b4b34a034336e328461c3
I20260924 09:55:58.990684 2277 tablet_service.cc:1514] Processing DeleteTablet for tablet 89ce407c649b4b34a034336e328461c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 09:55:58 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:39514
I20260924 09:55:58.990702 2416 tablet_service.cc:1514] Processing DeleteTablet for tablet 89ce407c649b4b34a034336e328461c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 09:55:58 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:34976
I20260924 09:55:58.990739 2555 tablet_service.cc:1514] Processing DeleteTablet for tablet 89ce407c649b4b34a034336e328461c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 09:55:58 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:44566
I20260924 09:55:58.991266 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu pid 2212
I20260924 09:55:59.003423 2805 tablet_replica.cc:330] T 89ce407c649b4b34a034336e328461c3 P 9ad27677628b497bb0f4e8636501851d: stopping tablet replica
I20260924 09:55:59.003813 2805 raft_consensus.cc:2229] T 89ce407c649b4b34a034336e328461c3 P 9ad27677628b497bb0f4e8636501851d [term 1 LEADER]: Raft consensus shutting down.
I20260924 09:55:59.004298 2805 raft_consensus.cc:2258] T 89ce407c649b4b34a034336e328461c3 P 9ad27677628b497bb0f4e8636501851d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 09:55:59.005427 2805 ts_tablet_manager.cc:1905] T 89ce407c649b4b34a034336e328461c3 P 9ad27677628b497bb0f4e8636501851d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 09:55:59.012750 2805 ts_tablet_manager.cc:1918] T 89ce407c649b4b34a034336e328461c3 P 9ad27677628b497bb0f4e8636501851d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 09:55:59.012966 2805 log.cc:1196] T 89ce407c649b4b34a034336e328461c3 P 9ad27677628b497bb0f4e8636501851d: Deleting WAL directory at /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790243735634835-388-0/minicluster-data/ts-0/wal/wals/89ce407c649b4b34a034336e328461c3
I20260924 09:55:59.013499 2805 ts_tablet_manager.cc:1939] T 89ce407c649b4b34a034336e328461c3 P 9ad27677628b497bb0f4e8636501851d: Deleting consensus metadata
I20260924 09:55:59.014695 2128 catalog_manager.cc:4841] TS 9ad27677628b497bb0f4e8636501851d (127.0.97.1:44953): tablet 89ce407c649b4b34a034336e328461c3 (table test-table [id=a25eb3ea885445168d29b622061b162b]) successfully deleted
I20260924 09:55:59.018373 2806 tablet_replica.cc:330] T 89ce407c649b4b34a034336e328461c3 P e4ce48a623fa4741bed2e12aa3b7a452: stopping tablet replica
I20260924 09:55:59.018586 2806 raft_consensus.cc:2229] T 89ce407c649b4b34a034336e328461c3 P e4ce48a623fa4741bed2e12aa3b7a452 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 09:55:59.018604 2807 tablet_replica.cc:330] T 89ce407c649b4b34a034336e328461c3 P ccac0f0c5aa54f1abe1d585545d2c34a: stopping tablet replica
I20260924 09:55:59.018788 2806 raft_consensus.cc:2258] T 89ce407c649b4b34a034336e328461c3 P e4ce48a623fa4741bed2e12aa3b7a452 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 09:55:59.018787 2807 raft_consensus.cc:2229] T 89ce407c649b4b34a034336e328461c3 P ccac0f0c5aa54f1abe1d585545d2c34a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 09:55:59.018987 2807 raft_consensus.cc:2258] T 89ce407c649b4b34a034336e328461c3 P ccac0f0c5aa54f1abe1d585545d2c34a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 09:55:59.019965 2807 ts_tablet_manager.cc:1905] T 89ce407c649b4b34a034336e328461c3 P ccac0f0c5aa54f1abe1d585545d2c34a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 09:55:59.020576 2806 ts_tablet_manager.cc:1905] T 89ce407c649b4b34a034336e328461c3 P e4ce48a623fa4741bed2e12aa3b7a452: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 09:55:59.025511 2806 ts_tablet_manager.cc:1918] T 89ce407c649b4b34a034336e328461c3 P e4ce48a623fa4741bed2e12aa3b7a452: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 09:55:59.025640 2806 log.cc:1196] T 89ce407c649b4b34a034336e328461c3 P e4ce48a623fa4741bed2e12aa3b7a452: Deleting WAL directory at /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790243735634835-388-0/minicluster-data/ts-2/wal/wals/89ce407c649b4b34a034336e328461c3
I20260924 09:55:59.025724 2807 ts_tablet_manager.cc:1918] T 89ce407c649b4b34a034336e328461c3 P ccac0f0c5aa54f1abe1d585545d2c34a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 09:55:59.025844 2807 log.cc:1196] T 89ce407c649b4b34a034336e328461c3 P ccac0f0c5aa54f1abe1d585545d2c34a: Deleting WAL directory at /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790243735634835-388-0/minicluster-data/ts-1/wal/wals/89ce407c649b4b34a034336e328461c3
I20260924 09:55:59.026089 2806 ts_tablet_manager.cc:1939] T 89ce407c649b4b34a034336e328461c3 P e4ce48a623fa4741bed2e12aa3b7a452: Deleting consensus metadata
Sep 24 09:55:59 dist-test-slave-w8d2 krb5kdc[2096](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243756, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260924 09:55:59.026190 2807 ts_tablet_manager.cc:1939] T 89ce407c649b4b34a034336e328461c3 P ccac0f0c5aa54f1abe1d585545d2c34a: Deleting consensus metadata
I20260924 09:55:59.027115 2125 catalog_manager.cc:4841] TS e4ce48a623fa4741bed2e12aa3b7a452 (127.0.97.3:42817): tablet 89ce407c649b4b34a034336e328461c3 (table test-table [id=a25eb3ea885445168d29b622061b162b]) successfully deleted
I20260924 09:55:59.027374 2125 catalog_manager.cc:4841] TS ccac0f0c5aa54f1abe1d585545d2c34a (127.0.97.2:35579): tablet 89ce407c649b4b34a034336e328461c3 (table test-table [id=a25eb3ea885445168d29b622061b162b]) successfully deleted
I20260924 09:55:59.032212 2809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:55:58.993362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39540 (local address 127.0.97.1:44953)
0924 09:55:59.017322 (+ 23960us) server_negotiation.cc:206] Beginning negotiation
0924 09:55:59.017328 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 09:55:59.017352 (+ 24us) server_negotiation.cc:407] Connection header received
0924 09:55:59.017408 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:55:59.017414 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:55:59.017494 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:55:59.017602 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:55:59.019134 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:55:59.020077 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:55:59.021732 (+ 1655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:55:59.022320 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:55:59.022447 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:55:59.027182 (+ 4735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:55:59.027197 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:55:59.027213 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:55:59.027285 (+ 72us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:55:59.029818 (+ 2533us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:55:59.030414 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:55:59.030421 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:55:59.030423 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:55:59.030553 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:55:59.030862 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:55:59.030865 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:55:59.030867 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:55:59.031365 (+ 498us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:55:59.031582 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0924 09:55:59.031796 (+ 214us) server_negotiation.cc:299] Negotiation successful
0924 09:55:59.031960 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23819,"thread_start_us":24559,"threads_started":1}
W20260924 09:55:59.032852 2342 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39540
W20260924 09:55:59.033427 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu pid 2212: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 09:55:59.033601 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu with pid 2212
I20260924 09:55:59.054420 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu pid 2351
Sep 24 09:55:59 dist-test-slave-w8d2 krb5kdc[2096](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243756, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260924 09:55:59.067916 2810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:55:59.055110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34996 (local address 127.0.97.2:35579)
0924 09:55:59.055561 (+ 451us) server_negotiation.cc:206] Beginning negotiation
0924 09:55:59.055573 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 09:55:59.055589 (+ 16us) server_negotiation.cc:407] Connection header received
0924 09:55:59.055640 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:55:59.055644 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:55:59.055708 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:55:59.055792 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:55:59.056876 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:55:59.057563 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:55:59.058792 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:55:59.059286 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:55:59.059396 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:55:59.062938 (+ 3542us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:55:59.062952 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:55:59.062981 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:55:59.063017 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:55:59.065589 (+ 2572us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:55:59.066102 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:55:59.066108 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:55:59.066111 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:55:59.066235 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:55:59.066493 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:55:59.066592 (+ 99us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:55:59.066595 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:55:59.066927 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:55:59.067138 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0924 09:55:59.067450 (+ 312us) server_negotiation.cc:299] Negotiation successful
0924 09:55:59.067599 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":356,"thread_start_us":245,"threads_started":1}
W20260924 09:55:59.068475 2481 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34996
W20260924 09:55:59.069007 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu pid 2351: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 09:55:59.069169 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu with pid 2351
I20260924 09:55:59.087334 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu pid 2490
Sep 24 09:55:59 dist-test-slave-w8d2 krb5kdc[2096](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243756, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260924 09:55:59.100694 2811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:55:59.087986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44576 (local address 127.0.97.3:42817)
0924 09:55:59.088444 (+ 458us) server_negotiation.cc:206] Beginning negotiation
0924 09:55:59.088448 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 09:55:59.088463 (+ 15us) server_negotiation.cc:407] Connection header received
0924 09:55:59.088513 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:55:59.088516 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:55:59.088580 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:55:59.088669 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:55:59.089799 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:55:59.090492 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:55:59.091734 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:55:59.092121 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:55:59.092212 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:55:59.095936 (+ 3724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:55:59.095949 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:55:59.095965 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:55:59.095995 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:55:59.098537 (+ 2542us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:55:59.099090 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:55:59.099096 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:55:59.099098 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:55:59.099209 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:55:59.099594 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:55:59.099598 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:55:59.099600 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:55:59.099899 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:55:59.100133 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0924 09:55:59.100290 (+ 157us) server_negotiation.cc:299] Negotiation successful
0924 09:55:59.100436 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":366,"thread_start_us":254,"threads_started":1}
W20260924 09:55:59.101269 2620 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44576
W20260924 09:55:59.101807 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu pid 2490: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 09:55:59.101926 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu with pid 2490
I20260924 09:55:59.118439 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu pid 2109
I20260924 09:55:59.119516 2181 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 09:55:59.231503 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu with pid 2109
2026-09-24T09:55:59Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (8070 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790243735634835-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 09:55:59 dist-test-slave-w8d2 krb5kdc[2815](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 09:55:59 dist-test-slave-w8d2 krb5kdc[2815](info): set up 2 sockets
Sep 24 09:55:59 dist-test-slave-w8d2 krb5kdc[2815](info): commencing operation
krb5kdc: starting...
W20260924 09:56:00.227171 388 mini_kdc.cc:121] Time spent starting KDC: real 0.957s user 0.000s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790243735634835-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790243735634835-388-0/krb5kdc/test-user.keytab.
Sep 24 09:56:00 dist-test-slave-w8d2 krb5kdc[2815](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243760, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T09:56:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T09:56:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790243735634835-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790243735634835-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260924 09:56:00.386695 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
/tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790243735634835-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790243735634835-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790243735634835-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790243735634835-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42233
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790243735634835-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:42233
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790243735634835-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790243735634835-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790243735634835-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 09:56:00.691058 2828 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 09:56:00.691417 2828 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 09:56:00.691560 2828 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 09:56:00.698614 2828 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 09:56:00.698715 2828 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 09:56:00.698768 2828 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 09:56:00.698843 2828 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 09:56:00.698904 2828 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 09:56:00.708614 2828 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790243735634835-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790243735634835-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:42233
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790243735634835-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42233
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790243735634835-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790243735634835-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 09:56:00.709892 2828 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 09:56:00.711161 2828 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 09:56:00.722249 2833 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:56:00.722187 2836 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:56:00.722227 2834 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 09:56:00.723061 2828 server_base.cc:1034] running on GCE node
I20260924 09:56:00.723801 2828 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 09:56:00.725119 2828 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 09:56:00.726305 2828 hybrid_clock.cc:648] HybridClock initialized: now 1790243760726250 us; error 42 us; skew 500 ppm
I20260924 09:56:00.726614 2828 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 09:56:00 dist-test-slave-w8d2 krb5kdc[2815](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243760, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 09:56:00.733763 2828 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260924 09:56:00.735159 2828 webserver.cc:460] Webserver started at http://127.0.97.62:40341/ using document root <none> and password file <none>
I20260924 09:56:00.735821 2828 fs_manager.cc:359] Metadata directory not provided
I20260924 09:56:00.735935 2828 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 09:56:00.736223 2828 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 09:56:00.739010 2828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790243735634835-388-0/minicluster-data/master-0/data/instance:
uuid: "8c37684eceb540099377694b2440f0bf"
format_stamp: "Formatted at 2026-09-24 09:56:00 on dist-test-slave-w8d2"
server_key: "8abe4d9858af9fccaad070fc6a908f87"
server_key_iv: "89b159bf1af0112a21c88247f8c831cc"
server_key_version: "encryptionkey@0"
I20260924 09:56:00.739749 2828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790243735634835-388-0/minicluster-data/master-0/wal/instance:
uuid: "8c37684eceb540099377694b2440f0bf"
format_stamp: "Formatted at 2026-09-24 09:56:00 on dist-test-slave-w8d2"
server_key: "8abe4d9858af9fccaad070fc6a908f87"
server_key_iv: "89b159bf1af0112a21c88247f8c831cc"
server_key_version: "encryptionkey@0"
I20260924 09:56:00.744860 2828 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260924 09:56:00.748996 2843 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 09:56:00.750277 2828 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 09:56:00.750438 2828 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790243735634835-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790243735634835-388-0/minicluster-data/master-0/wal
uuid: "8c37684eceb540099377694b2440f0bf"
format_stamp: "Formatted at 2026-09-24 09:56:00 on dist-test-slave-w8d2"
server_key: "8abe4d9858af9fccaad070fc6a908f87"
server_key_iv: "89b159bf1af0112a21c88247f8c831cc"
server_key_version: "encryptionkey@0"
I20260924 09:56:00.750612 2828 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790243735634835-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790243735634835-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790243735634835-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 09:56:00.763844 2828 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 09:56:00.767732 2828 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 09:56:00.767973 2828 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 09:56:00.784090 2828 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:42233
I20260924 09:56:00.784109 2894 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:42233 every 8 connection(s)
I20260924 09:56:00.785450 2828 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790243735634835-388-0/minicluster-data/master-0/data/info.pb
I20260924 09:56:00.789070 2895 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 09:56:00.792855 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu as pid 2828
I20260924 09:56:00.793035 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790243735634835-388-0/minicluster-data/master-0/wal/instance
I20260924 09:56:00.793596 388 external_mini_cluster.cc:1428] Setting key a09467b27285b5e680fa5ad640baa5ad
I20260924 09:56:00.800660 2895 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c37684eceb540099377694b2440f0bf: Bootstrap starting.
I20260924 09:56:00.804145 2895 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8c37684eceb540099377694b2440f0bf: Neither blocks nor log segments found. Creating new log.
I20260924 09:56:00.805385 2895 log.cc:824] T 00000000000000000000000000000000 P 8c37684eceb540099377694b2440f0bf: Log is configured to *not* fsync() on all Append() calls
I20260924 09:56:00.808144 2895 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c37684eceb540099377694b2440f0bf: No bootstrap required, opened a new log
Sep 24 09:56:00 dist-test-slave-w8d2 krb5kdc[2815](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243760, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 09:56:00.816190 2898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:00.795456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39732 (local address 127.0.97.62:42233)
0924 09:56:00.796213 (+ 757us) server_negotiation.cc:206] Beginning negotiation
0924 09:56:00.796228 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 09:56:00.796278 (+ 50us) server_negotiation.cc:407] Connection header received
0924 09:56:00.799003 (+ 2725us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:56:00.799031 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:56:00.799537 (+ 506us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:56:00.799953 (+ 416us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:56:00.801108 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:00.803040 (+ 1932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:00.804243 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:00.805231 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:00.805415 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:00.809412 (+ 3997us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:56:00.809420 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:56:00.809434 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:56:00.809471 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:56:00.812355 (+ 2884us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:56:00.813137 (+ 782us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:56:00.813142 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:56:00.813148 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:56:00.813305 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:56:00.813661 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:56:00.813667 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:56:00.813669 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:56:00.814156 (+ 487us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:56:00.814358 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0924 09:56:00.814753 (+ 395us) server_negotiation.cc:299] Negotiation successful
0924 09:56:00.815144 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":529,"thread_start_us":321,"threads_started":1}
I20260924 09:56:00.816648 2895 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8c37684eceb540099377694b2440f0bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c37684eceb540099377694b2440f0bf" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42233 } }
I20260924 09:56:00.816980 2895 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8c37684eceb540099377694b2440f0bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 09:56:00.817086 2895 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8c37684eceb540099377694b2440f0bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c37684eceb540099377694b2440f0bf, State: Initialized, Role: FOLLOWER
I20260924 09:56:00.817672 2895 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8c37684eceb540099377694b2440f0bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c37684eceb540099377694b2440f0bf" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42233 } }
I20260924 09:56:00.817951 2895 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8c37684eceb540099377694b2440f0bf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 09:56:00.818075 2895 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8c37684eceb540099377694b2440f0bf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 09:56:00.818251 2895 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8c37684eceb540099377694b2440f0bf [term 0 FOLLOWER]: Advancing to term 1
I20260924 09:56:00.820447 2895 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8c37684eceb540099377694b2440f0bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c37684eceb540099377694b2440f0bf" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42233 } }
I20260924 09:56:00.820844 2895 leader_election.cc:302] T 00000000000000000000000000000000 P 8c37684eceb540099377694b2440f0bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8c37684eceb540099377694b2440f0bf; no voters:
I20260924 09:56:00.821326 2895 leader_election.cc:288] T 00000000000000000000000000000000 P 8c37684eceb540099377694b2440f0bf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 09:56:00.821429 2900 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8c37684eceb540099377694b2440f0bf [term 1 FOLLOWER]: Leader election won for term 1
I20260924 09:56:00.821993 2900 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8c37684eceb540099377694b2440f0bf [term 1 LEADER]: Becoming Leader. State: Replica: 8c37684eceb540099377694b2440f0bf, State: Running, Role: LEADER
I20260924 09:56:00.822494 2900 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8c37684eceb540099377694b2440f0bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c37684eceb540099377694b2440f0bf" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42233 } }
I20260924 09:56:00.823144 2895 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8c37684eceb540099377694b2440f0bf [sys.catalog]: configured and running, proceeding with master startup.
I20260924 09:56:00.825101 2902 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c37684eceb540099377694b2440f0bf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8c37684eceb540099377694b2440f0bf. Latest consensus state: current_term: 1 leader_uuid: "8c37684eceb540099377694b2440f0bf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c37684eceb540099377694b2440f0bf" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42233 } } }
I20260924 09:56:00.825172 2901 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c37684eceb540099377694b2440f0bf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8c37684eceb540099377694b2440f0bf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c37684eceb540099377694b2440f0bf" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42233 } } }
I20260924 09:56:00.825500 2902 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c37684eceb540099377694b2440f0bf [sys.catalog]: This master's current role is: LEADER
I20260924 09:56:00.825506 2901 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c37684eceb540099377694b2440f0bf [sys.catalog]: This master's current role is: LEADER
I20260924 09:56:00.826427 2906 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 09:56:00.831645 2906 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 09:56:00.840605 2906 catalog_manager.cc:1347] Generated new cluster ID: c907a4e10f87465e94b67437bfdf527a
I20260924 09:56:00.840713 2906 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 09:56:00.886015 2906 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 09:56:00.887898 2906 catalog_manager.cc:1504] Loading token signing keys...
I20260924 09:56:00.899832 2906 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8c37684eceb540099377694b2440f0bf: Generated new TSK 0
I20260924 09:56:00.900662 2906 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790243735634835-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790243735634835-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260924 09:56:00.956494 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
/tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790243735634835-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790243735634835-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790243735634835-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790243735634835-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42233
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790243735634835-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790243735634835-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790243735634835-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790243735634835-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 09:56:01.277506 2921 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 09:56:01.277765 2921 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 09:56:01.277855 2921 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 09:56:01.284826 2921 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 09:56:01.284922 2921 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 09:56:01.285040 2921 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260924 09:56:01.294301 2921 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790243735634835-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790243735634835-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790243735634835-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790243735634835-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42233
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790243735634835-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 09:56:01.295472 2921 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 09:56:01.296643 2921 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 09:56:01.308687 2927 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:56:01.308722 2929 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:56:01.308904 2926 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 09:56:01.309520 2921 server_base.cc:1034] running on GCE node
I20260924 09:56:01.310068 2921 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 09:56:01.310961 2921 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 09:56:01.312148 2921 hybrid_clock.cc:648] HybridClock initialized: now 1790243761312100 us; error 36 us; skew 500 ppm
I20260924 09:56:01.312472 2921 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 09:56:01 dist-test-slave-w8d2 krb5kdc[2815](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243761, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 09:56:01.315425 2921 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260924 09:56:01.316743 2921 webserver.cc:460] Webserver started at http://127.0.97.1:40957/ using document root <none> and password file <none>
I20260924 09:56:01.317412 2921 fs_manager.cc:359] Metadata directory not provided
I20260924 09:56:01.317520 2921 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 09:56:01.317803 2921 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 09:56:01.320544 2921 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790243735634835-388-0/minicluster-data/ts-0/data/instance:
uuid: "dfa50b3064254790bda4d7278160d47c"
format_stamp: "Formatted at 2026-09-24 09:56:01 on dist-test-slave-w8d2"
server_key: "b18e8626a74de4c447494d96fcbb0ba6"
server_key_iv: "1690af2f775b5e2d9b193ee683326947"
server_key_version: "encryptionkey@0"
I20260924 09:56:01.321261 2921 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790243735634835-388-0/minicluster-data/ts-0/wal/instance:
uuid: "dfa50b3064254790bda4d7278160d47c"
format_stamp: "Formatted at 2026-09-24 09:56:01 on dist-test-slave-w8d2"
server_key: "b18e8626a74de4c447494d96fcbb0ba6"
server_key_iv: "1690af2f775b5e2d9b193ee683326947"
server_key_version: "encryptionkey@0"
I20260924 09:56:01.326108 2921 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 09:56:01.329450 2936 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 09:56:01.330734 2921 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260924 09:56:01.330880 2921 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790243735634835-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790243735634835-388-0/minicluster-data/ts-0/wal
uuid: "dfa50b3064254790bda4d7278160d47c"
format_stamp: "Formatted at 2026-09-24 09:56:01 on dist-test-slave-w8d2"
server_key: "b18e8626a74de4c447494d96fcbb0ba6"
server_key_iv: "1690af2f775b5e2d9b193ee683326947"
server_key_version: "encryptionkey@0"
I20260924 09:56:01.331063 2921 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790243735634835-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790243735634835-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790243735634835-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 09:56:01.374717 2921 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 09:56:01.378263 2921 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 09:56:01.378526 2921 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 09:56:01.379419 2921 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 09:56:01.380645 2921 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 09:56:01.380717 2921 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 09:56:01.380826 2921 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 09:56:01.380883 2921 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 09:56:01.408838 2921 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:40775
I20260924 09:56:01.408883 3048 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:40775 every 8 connection(s)
I20260924 09:56:01.410005 2921 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790243735634835-388-0/minicluster-data/ts-0/data/info.pb
I20260924 09:56:01.414812 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu as pid 2921
I20260924 09:56:01.414991 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790243735634835-388-0/minicluster-data/ts-0/wal/instance
I20260924 09:56:01.415771 388 external_mini_cluster.cc:1428] Setting key 9ba4ac0c8d67ceee6d6367bcd691218c
Sep 24 09:56:01 dist-test-slave-w8d2 krb5kdc[2815](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243761, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 09:56:01.430104 3051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:01.412337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:46579 (local address 127.0.97.62:42233)
0924 09:56:01.412729 (+ 392us) server_negotiation.cc:206] Beginning negotiation
0924 09:56:01.412733 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 09:56:01.413649 (+ 916us) server_negotiation.cc:407] Connection header received
0924 09:56:01.415851 (+ 2202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:56:01.415857 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:56:01.415931 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:56:01.416024 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:56:01.417662 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:01.418358 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:01.420452 (+ 2094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:01.420825 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:01.420916 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:01.424413 (+ 3497us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:56:01.424420 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:56:01.424424 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:56:01.424461 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:56:01.427314 (+ 2853us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:56:01.427939 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:56:01.427951 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:56:01.427954 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:56:01.428078 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:56:01.428456 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:56:01.428461 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:56:01.428464 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:56:01.428750 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:56:01.428883 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0924 09:56:01.429540 (+ 657us) server_negotiation.cc:299] Negotiation successful
0924 09:56:01.429739 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":297,"thread_start_us":207,"threads_started":1}
I20260924 09:56:01.430784 3052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:01.412854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42233 (local address 127.0.97.1:46579)
0924 09:56:01.413498 (+ 644us) negotiation.cc:107] Waiting for socket to connect
0924 09:56:01.413533 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 09:56:01.415554 (+ 2021us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:56:01.416281 (+ 727us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:56:01.416293 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:56:01.416820 (+ 527us) client_negotiation.cc:189] Negotiated authn=SASL
0924 09:56:01.417519 (+ 699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:01.417537 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:01.418513 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:01.418518 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:01.420329 (+ 1811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:01.420342 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:01.421015 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:01.421021 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:01.421274 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:01.421903 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 09:56:01.421929 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0924 09:56:01.424230 (+ 2301us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 09:56:01.427431 (+ 3201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 09:56:01.427437 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 09:56:01.427448 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0924 09:56:01.427828 (+ 380us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 09:56:01.428162 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 09:56:01.428165 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 09:56:01.428167 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 09:56:01.428360 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 09:56:01.428847 (+ 487us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 09:56:01.428852 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 09:56:01.429183 (+ 331us) client_negotiation.cc:769] Sending connection context
0924 09:56:01.429413 (+ 230us) client_negotiation.cc:240] Negotiation successful
0924 09:56:01.429774 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":246,"threads_started":1}
I20260924 09:56:01.432221 3049 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42233
I20260924 09:56:01.432507 3049 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260924 09:56:01.433130 3049 heartbeater.cc:511] Master 127.0.97.62:42233 requested a full tablet report, sending...
I20260924 09:56:01.435014 2860 ts_manager.cc:194] Registered new tserver with Master: dfa50b3064254790bda4d7278160d47c (127.0.97.1:40775)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260924 09:56:01.436885 2860 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:46579
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790243735634835-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790243735634835-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260924 09:56:01.453529 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
/tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790243735634835-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790243735634835-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790243735634835-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790243735634835-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42233
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790243735634835-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790243735634835-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790243735634835-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790243735634835-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 09:56:01.742960 3055 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 09:56:01.743222 3055 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 09:56:01.743377 3055 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 09:56:01.750202 3055 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 09:56:01.750305 3055 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 09:56:01.750447 3055 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260924 09:56:01.759521 3055 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790243735634835-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790243735634835-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790243735634835-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790243735634835-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42233
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790243735634835-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 09:56:01.760655 3055 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 09:56:01.761904 3055 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 09:56:01.773895 3060 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:56:01.773890 3061 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:56:01.773926 3063 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 09:56:01.774664 3055 server_base.cc:1034] running on GCE node
I20260924 09:56:01.775197 3055 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 09:56:01.776590 3055 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 09:56:01.777776 3055 hybrid_clock.cc:648] HybridClock initialized: now 1790243761777722 us; error 52 us; skew 500 ppm
I20260924 09:56:01.778056 3055 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 09:56:01 dist-test-slave-w8d2 krb5kdc[2815](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243761, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 09:56:01.780819 3055 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260924 09:56:01.782037 3055 webserver.cc:460] Webserver started at http://127.0.97.2:34439/ using document root <none> and password file <none>
I20260924 09:56:01.782639 3055 fs_manager.cc:359] Metadata directory not provided
I20260924 09:56:01.782744 3055 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 09:56:01.783013 3055 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 09:56:01.785763 3055 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790243735634835-388-0/minicluster-data/ts-1/data/instance:
uuid: "039f03ab9ab5491aa8ebf148f8c45ec7"
format_stamp: "Formatted at 2026-09-24 09:56:01 on dist-test-slave-w8d2"
server_key: "821fbe5735dfcc401bcde83c3dc21385"
server_key_iv: "8ed3b9353d693c01cc8696d9f84a42b9"
server_key_version: "encryptionkey@0"
I20260924 09:56:01.786501 3055 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790243735634835-388-0/minicluster-data/ts-1/wal/instance:
uuid: "039f03ab9ab5491aa8ebf148f8c45ec7"
format_stamp: "Formatted at 2026-09-24 09:56:01 on dist-test-slave-w8d2"
server_key: "821fbe5735dfcc401bcde83c3dc21385"
server_key_iv: "8ed3b9353d693c01cc8696d9f84a42b9"
server_key_version: "encryptionkey@0"
I20260924 09:56:01.791292 3055 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260924 09:56:01.794494 3070 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 09:56:01.795782 3055 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 09:56:01.795917 3055 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790243735634835-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790243735634835-388-0/minicluster-data/ts-1/wal
uuid: "039f03ab9ab5491aa8ebf148f8c45ec7"
format_stamp: "Formatted at 2026-09-24 09:56:01 on dist-test-slave-w8d2"
server_key: "821fbe5735dfcc401bcde83c3dc21385"
server_key_iv: "8ed3b9353d693c01cc8696d9f84a42b9"
server_key_version: "encryptionkey@0"
I20260924 09:56:01.796092 3055 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790243735634835-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790243735634835-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790243735634835-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 09:56:01.808755 3055 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 09:56:01.812093 3055 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 09:56:01.812322 3055 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 09:56:01.813122 3055 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 09:56:01.814244 3055 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 09:56:01.814313 3055 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 09:56:01.814414 3055 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 09:56:01.814466 3055 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 09:56:01.842041 3055 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:38845
I20260924 09:56:01.842073 3182 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:38845 every 8 connection(s)
I20260924 09:56:01.843171 3055 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790243735634835-388-0/minicluster-data/ts-1/data/info.pb
I20260924 09:56:01.846427 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu as pid 3055
I20260924 09:56:01.846575 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790243735634835-388-0/minicluster-data/ts-1/wal/instance
I20260924 09:56:01.847040 388 external_mini_cluster.cc:1428] Setting key a835947d1ff5e66a31e7c21617e839af
Sep 24 09:56:01 dist-test-slave-w8d2 krb5kdc[2815](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243761, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 09:56:01.861479 3051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:01.845569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:36501 (local address 127.0.97.62:42233)
0924 09:56:01.845730 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0924 09:56:01.845734 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 09:56:01.846846 (+ 1112us) server_negotiation.cc:407] Connection header received
0924 09:56:01.848640 (+ 1794us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:56:01.848645 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:56:01.848728 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:56:01.848807 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:56:01.850163 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:01.850839 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:01.852337 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:01.852732 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:01.852837 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:01.856340 (+ 3503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:56:01.856344 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:56:01.856348 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:56:01.856376 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:56:01.858601 (+ 2225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:56:01.859432 (+ 831us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:56:01.859435 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:56:01.859438 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:56:01.859570 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:56:01.860006 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:56:01.860009 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:56:01.860011 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:56:01.860235 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:56:01.860333 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0924 09:56:01.861099 (+ 766us) server_negotiation.cc:299] Negotiation successful
0924 09:56:01.861248 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260924 09:56:01.862794 3185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:01.845959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42233 (local address 127.0.97.2:36501)
0924 09:56:01.846695 (+ 736us) negotiation.cc:107] Waiting for socket to connect
0924 09:56:01.846742 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0924 09:56:01.848398 (+ 1656us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:56:01.849031 (+ 633us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:56:01.849040 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:56:01.849504 (+ 464us) client_negotiation.cc:189] Negotiated authn=SASL
0924 09:56:01.850012 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:01.850029 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:01.850971 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:01.850976 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:01.852210 (+ 1234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:01.852221 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:01.852923 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:01.852930 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:01.853128 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:01.853783 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 09:56:01.853804 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 09:56:01.856143 (+ 2339us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 09:56:01.858733 (+ 2590us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 09:56:01.858742 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 09:56:01.858756 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 09:56:01.859330 (+ 574us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 09:56:01.859665 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 09:56:01.859670 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 09:56:01.859674 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 09:56:01.859910 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 09:56:01.860343 (+ 433us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 09:56:01.860351 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 09:56:01.860746 (+ 395us) client_negotiation.cc:769] Sending connection context
0924 09:56:01.861038 (+ 292us) client_negotiation.cc:240] Negotiation successful
0924 09:56:01.861524 (+ 486us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":303,"threads_started":1}
I20260924 09:56:01.864100 3183 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42233
I20260924 09:56:01.864387 3183 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260924 09:56:01.865123 3183 heartbeater.cc:511] Master 127.0.97.62:42233 requested a full tablet report, sending...
I20260924 09:56:01.866544 2860 ts_manager.cc:194] Registered new tserver with Master: 039f03ab9ab5491aa8ebf148f8c45ec7 (127.0.97.2:38845)
I20260924 09:56:01.867549 2860 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:36501
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790243735634835-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790243735634835-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260924 09:56:01.883997 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
/tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790243735634835-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790243735634835-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790243735634835-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790243735634835-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42233
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790243735634835-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790243735634835-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790243735634835-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790243735634835-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 09:56:02.181277 3188 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 09:56:02.181547 3188 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 09:56:02.181666 3188 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 09:56:02.188608 3188 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 09:56:02.188714 3188 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 09:56:02.188862 3188 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260924 09:56:02.197953 3188 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790243735634835-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790243735634835-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790243735634835-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790243735634835-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42233
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790243735634835-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 09:56:02.199111 3188 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 09:56:02.200407 3188 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 09:56:02.212111 3193 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:56:02.212134 3194 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:56:02.212066 3196 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 09:56:02.212845 3188 server_base.cc:1034] running on GCE node
I20260924 09:56:02.213352 3188 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 09:56:02.214160 3188 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 09:56:02.215310 3188 hybrid_clock.cc:648] HybridClock initialized: now 1790243762215260 us; error 39 us; skew 500 ppm
I20260924 09:56:02.215610 3188 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 09:56:02 dist-test-slave-w8d2 krb5kdc[2815](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243762, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 09:56:02.218391 3188 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260924 09:56:02.219684 3188 webserver.cc:460] Webserver started at http://127.0.97.3:43561/ using document root <none> and password file <none>
I20260924 09:56:02.220273 3188 fs_manager.cc:359] Metadata directory not provided
I20260924 09:56:02.220386 3188 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 09:56:02.220654 3188 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 09:56:02.223340 3188 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790243735634835-388-0/minicluster-data/ts-2/data/instance:
uuid: "ccf55552d76e4d1c850c73090c87187d"
format_stamp: "Formatted at 2026-09-24 09:56:02 on dist-test-slave-w8d2"
server_key: "cbdc93c977f92fb31a5cb87a6b30b3c2"
server_key_iv: "72b15e766415497eb7d8ad7edf9bcd95"
server_key_version: "encryptionkey@0"
I20260924 09:56:02.224066 3188 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790243735634835-388-0/minicluster-data/ts-2/wal/instance:
uuid: "ccf55552d76e4d1c850c73090c87187d"
format_stamp: "Formatted at 2026-09-24 09:56:02 on dist-test-slave-w8d2"
server_key: "cbdc93c977f92fb31a5cb87a6b30b3c2"
server_key_iv: "72b15e766415497eb7d8ad7edf9bcd95"
server_key_version: "encryptionkey@0"
I20260924 09:56:02.228811 3188 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 09:56:02.231957 3203 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 09:56:02.233175 3188 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 09:56:02.233312 3188 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790243735634835-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790243735634835-388-0/minicluster-data/ts-2/wal
uuid: "ccf55552d76e4d1c850c73090c87187d"
format_stamp: "Formatted at 2026-09-24 09:56:02 on dist-test-slave-w8d2"
server_key: "cbdc93c977f92fb31a5cb87a6b30b3c2"
server_key_iv: "72b15e766415497eb7d8ad7edf9bcd95"
server_key_version: "encryptionkey@0"
I20260924 09:56:02.233503 3188 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790243735634835-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790243735634835-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790243735634835-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 09:56:02.252238 3188 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 09:56:02.255725 3188 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 09:56:02.255964 3188 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 09:56:02.256825 3188 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 09:56:02.258038 3188 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 09:56:02.258111 3188 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 09:56:02.258224 3188 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 09:56:02.258273 3188 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 09:56:02.286193 3188 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:38643
I20260924 09:56:02.286233 3315 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:38643 every 8 connection(s)
I20260924 09:56:02.287443 3188 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790243735634835-388-0/minicluster-data/ts-2/data/info.pb
I20260924 09:56:02.287891 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu as pid 3188
I20260924 09:56:02.288036 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790243735634835-388-0/minicluster-data/ts-2/wal/instance
I20260924 09:56:02.288570 388 external_mini_cluster.cc:1428] Setting key e1f6b9e35dd3059930769250411a99e8
Sep 24 09:56:02 dist-test-slave-w8d2 krb5kdc[2815](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243762, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 09:56:02.305362 3051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:02.289836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:43049 (local address 127.0.97.62:42233)
0924 09:56:02.290013 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0924 09:56:02.290019 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 09:56:02.291320 (+ 1301us) server_negotiation.cc:407] Connection header received
0924 09:56:02.293064 (+ 1744us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:56:02.293068 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:56:02.293129 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:56:02.293209 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:56:02.294644 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:02.295424 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:02.296840 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:02.297203 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:02.297285 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:02.300327 (+ 3042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:56:02.300331 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:56:02.300334 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:56:02.300363 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:56:02.302887 (+ 2524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:56:02.303635 (+ 748us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:56:02.303639 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:56:02.303641 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:56:02.303747 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:56:02.304062 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:56:02.304065 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:56:02.304067 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:56:02.304298 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:56:02.304394 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0924 09:56:02.304974 (+ 580us) server_negotiation.cc:299] Negotiation successful
0924 09:56:02.305123 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 09:56:02.306171 3318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:02.290398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42233 (local address 127.0.97.3:43049)
0924 09:56:02.291152 (+ 754us) negotiation.cc:107] Waiting for socket to connect
0924 09:56:02.291195 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 09:56:02.292814 (+ 1619us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:56:02.293397 (+ 583us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:56:02.293405 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:56:02.293993 (+ 588us) client_negotiation.cc:189] Negotiated authn=SASL
0924 09:56:02.294523 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:02.294541 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:02.295530 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:02.295534 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:02.296737 (+ 1203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:02.296748 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:02.297354 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:02.297360 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:02.297540 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:02.298083 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 09:56:02.298106 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 09:56:02.300164 (+ 2058us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 09:56:02.303000 (+ 2836us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 09:56:02.303008 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 09:56:02.303022 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 09:56:02.303521 (+ 499us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 09:56:02.303825 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 09:56:02.303828 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 09:56:02.303830 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 09:56:02.303984 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 09:56:02.304377 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 09:56:02.304381 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 09:56:02.304659 (+ 278us) client_negotiation.cc:769] Sending connection context
0924 09:56:02.304859 (+ 200us) client_negotiation.cc:240] Negotiation successful
0924 09:56:02.305197 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":304,"threads_started":1}
I20260924 09:56:02.307312 3316 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42233
I20260924 09:56:02.307577 3316 heartbeater.cc:464] Registering TS with master...
I20260924 09:56:02.308243 3316 heartbeater.cc:511] Master 127.0.97.62:42233 requested a full tablet report, sending...
I20260924 09:56:02.309480 2860 ts_manager.cc:194] Registered new tserver with Master: ccf55552d76e4d1c850c73090c87187d (127.0.97.3:38643)
I20260924 09:56:02.310228 2860 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:43049
I20260924 09:56:02.315469 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260924 09:56:02.341254 3051 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 09:56:02.339173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39742 (local address 127.0.97.62:42233)
0924 09:56:02.339348 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0924 09:56:02.339353 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 09:56:02.339704 (+ 351us) server_negotiation.cc:407] Connection header received
0924 09:56:02.339844 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:56:02.339848 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:56:02.339909 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:56:02.340004 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:56:02.341048 (+ 1044us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39742: BlockingRecv error: recv got EOF from 127.0.0.1:39742 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
I20260924 09:56:02.344061 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu pid 2921
W20260924 09:56:02.346298 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu pid 2921: Not authorized: Client connection negotiation failed: client connection to 127.0.97.1:40775: server requires authentication, but client does not have Kerberos credentials available
I20260924 09:56:02.346453 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu with pid 2921
I20260924 09:56:02.358173 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu pid 3055
W20260924 09:56:02.360605 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu pid 3055: Not authorized: Client connection negotiation failed: client connection to 127.0.97.2:38845: server requires authentication, but client does not have Kerberos credentials available
I20260924 09:56:02.360736 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu with pid 3055
I20260924 09:56:02.372224 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu pid 3188
W20260924 09:56:02.374751 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu pid 3188: Not authorized: Client connection negotiation failed: client connection to 127.0.97.3:38643: server requires authentication, but client does not have Kerberos credentials available
I20260924 09:56:02.374882 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu with pid 3188
I20260924 09:56:02.386274 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu pid 2828
I20260924 09:56:02.387359 2890 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 09:56:02.485512 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu with pid 2828
2026-09-24T09:56:02Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (3251 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790243735634835-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 09:56:02 dist-test-slave-w8d2 krb5kdc[3333](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 09:56:02 dist-test-slave-w8d2 krb5kdc[3333](info): set up 2 sockets
Sep 24 09:56:02 dist-test-slave-w8d2 krb5kdc[3333](info): commencing operation
krb5kdc: starting...
W20260924 09:56:03.475283 388 mini_kdc.cc:121] Time spent starting KDC: real 0.954s user 0.003s sys 0.011s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790243735634835-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790243735634835-388-0/krb5kdc/test-user.keytab.
Sep 24 09:56:03 dist-test-slave-w8d2 krb5kdc[3333](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243763, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T09:56:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T09:56:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790243735634835-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790243735634835-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260924 09:56:03.636615 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
/tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790243735634835-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790243735634835-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790243735634835-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790243735634835-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46231
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790243735634835-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:46231
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790243735634835-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790243735634835-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790243735634835-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 09:56:03.932524 3346 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 09:56:03.932782 3346 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 09:56:03.932876 3346 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 09:56:03.939672 3346 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 09:56:03.939765 3346 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 09:56:03.939814 3346 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 09:56:03.939867 3346 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 09:56:03.939913 3346 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 09:56:03.949129 3346 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790243735634835-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790243735634835-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:46231
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790243735634835-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46231
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790243735634835-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790243735634835-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 09:56:03.950314 3346 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 09:56:03.951546 3346 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 09:56:03.962002 3352 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:56:03.962080 3351 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:56:03.962052 3354 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 09:56:03.962983 3346 server_base.cc:1034] running on GCE node
I20260924 09:56:03.963774 3346 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 09:56:03.965080 3346 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 09:56:03.966272 3346 hybrid_clock.cc:648] HybridClock initialized: now 1790243763966228 us; error 42 us; skew 500 ppm
I20260924 09:56:03.966583 3346 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 09:56:03 dist-test-slave-w8d2 krb5kdc[3333](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243763, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 09:56:03.969429 3346 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260924 09:56:03.970788 3346 webserver.cc:460] Webserver started at http://127.0.97.62:45737/ using document root <none> and password file <none>
I20260924 09:56:03.971464 3346 fs_manager.cc:359] Metadata directory not provided
I20260924 09:56:03.971573 3346 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 09:56:03.971855 3346 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 09:56:03.974586 3346 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790243735634835-388-0/minicluster-data/master-0/data/instance:
uuid: "6689502fe43b4a2f9ab26aa2f7e9dc0d"
format_stamp: "Formatted at 2026-09-24 09:56:03 on dist-test-slave-w8d2"
server_key: "192f63eaa1fb6fabf38f0ade5e1446b4"
server_key_iv: "593e06d7bf731f4fa11c15229a3eb987"
server_key_version: "encryptionkey@0"
I20260924 09:56:03.975309 3346 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790243735634835-388-0/minicluster-data/master-0/wal/instance:
uuid: "6689502fe43b4a2f9ab26aa2f7e9dc0d"
format_stamp: "Formatted at 2026-09-24 09:56:03 on dist-test-slave-w8d2"
server_key: "192f63eaa1fb6fabf38f0ade5e1446b4"
server_key_iv: "593e06d7bf731f4fa11c15229a3eb987"
server_key_version: "encryptionkey@0"
I20260924 09:56:03.979857 3346 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.002s
I20260924 09:56:03.982852 3361 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 09:56:03.984063 3346 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260924 09:56:03.984203 3346 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790243735634835-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790243735634835-388-0/minicluster-data/master-0/wal
uuid: "6689502fe43b4a2f9ab26aa2f7e9dc0d"
format_stamp: "Formatted at 2026-09-24 09:56:03 on dist-test-slave-w8d2"
server_key: "192f63eaa1fb6fabf38f0ade5e1446b4"
server_key_iv: "593e06d7bf731f4fa11c15229a3eb987"
server_key_version: "encryptionkey@0"
I20260924 09:56:03.984362 3346 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790243735634835-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790243735634835-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790243735634835-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 09:56:03.998796 3346 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 09:56:04.002609 3346 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 09:56:04.002835 3346 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 09:56:04.018508 3346 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:46231
I20260924 09:56:04.018530 3412 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:46231 every 8 connection(s)
I20260924 09:56:04.019721 3346 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790243735634835-388-0/minicluster-data/master-0/data/info.pb
I20260924 09:56:04.020735 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu as pid 3346
I20260924 09:56:04.020937 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790243735634835-388-0/minicluster-data/master-0/wal/instance
I20260924 09:56:04.021536 388 external_mini_cluster.cc:1428] Setting key 330549c08bd14581d9a520f4743e6c9e
I20260924 09:56:04.023649 3413 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 24 09:56:04 dist-test-slave-w8d2 krb5kdc[3333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243763, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 09:56:04.035583 3413 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6689502fe43b4a2f9ab26aa2f7e9dc0d: Bootstrap starting.
I20260924 09:56:04.039461 3413 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6689502fe43b4a2f9ab26aa2f7e9dc0d: Neither blocks nor log segments found. Creating new log.
I20260924 09:56:04.040567 3413 log.cc:824] T 00000000000000000000000000000000 P 6689502fe43b4a2f9ab26aa2f7e9dc0d: Log is configured to *not* fsync() on all Append() calls
I20260924 09:56:04.043138 3416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:04.023129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43940 (local address 127.0.97.62:46231)
0924 09:56:04.023870 (+ 741us) server_negotiation.cc:206] Beginning negotiation
0924 09:56:04.023882 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 09:56:04.023924 (+ 42us) server_negotiation.cc:407] Connection header received
0924 09:56:04.025430 (+ 1506us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:56:04.025449 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:56:04.025811 (+ 362us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:56:04.026118 (+ 307us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:56:04.027770 (+ 1652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:04.029077 (+ 1307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:04.030713 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:04.031352 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:04.031474 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:04.035617 (+ 4143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:56:04.035628 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:56:04.035654 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:56:04.035701 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:56:04.039169 (+ 3468us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:56:04.039808 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:56:04.039816 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:56:04.039824 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:56:04.040012 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:56:04.040315 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:56:04.040320 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:56:04.040323 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:56:04.040956 (+ 633us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:56:04.041246 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0924 09:56:04.041596 (+ 350us) server_negotiation.cc:299] Negotiation successful
0924 09:56:04.041965 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":550,"thread_start_us":323,"threads_started":1}
I20260924 09:56:04.043591 3413 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6689502fe43b4a2f9ab26aa2f7e9dc0d: No bootstrap required, opened a new log
I20260924 09:56:04.046581 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 09:56:04.050783 3413 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6689502fe43b4a2f9ab26aa2f7e9dc0d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6689502fe43b4a2f9ab26aa2f7e9dc0d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46231 } }
I20260924 09:56:04.051091 3413 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6689502fe43b4a2f9ab26aa2f7e9dc0d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 09:56:04.051190 3413 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6689502fe43b4a2f9ab26aa2f7e9dc0d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6689502fe43b4a2f9ab26aa2f7e9dc0d, State: Initialized, Role: FOLLOWER
I20260924 09:56:04.051743 3413 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6689502fe43b4a2f9ab26aa2f7e9dc0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6689502fe43b4a2f9ab26aa2f7e9dc0d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46231 } }
I20260924 09:56:04.051944 3413 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6689502fe43b4a2f9ab26aa2f7e9dc0d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 09:56:04.052064 3413 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6689502fe43b4a2f9ab26aa2f7e9dc0d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 09:56:04.052218 3413 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6689502fe43b4a2f9ab26aa2f7e9dc0d [term 0 FOLLOWER]: Advancing to term 1
I20260924 09:56:04.054150 3413 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6689502fe43b4a2f9ab26aa2f7e9dc0d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6689502fe43b4a2f9ab26aa2f7e9dc0d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46231 } }
I20260924 09:56:04.054581 3413 leader_election.cc:302] T 00000000000000000000000000000000 P 6689502fe43b4a2f9ab26aa2f7e9dc0d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6689502fe43b4a2f9ab26aa2f7e9dc0d; no voters:
I20260924 09:56:04.055146 3413 leader_election.cc:288] T 00000000000000000000000000000000 P 6689502fe43b4a2f9ab26aa2f7e9dc0d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 09:56:04.055272 3419 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6689502fe43b4a2f9ab26aa2f7e9dc0d [term 1 FOLLOWER]: Leader election won for term 1
I20260924 09:56:04.056285 3419 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6689502fe43b4a2f9ab26aa2f7e9dc0d [term 1 LEADER]: Becoming Leader. State: Replica: 6689502fe43b4a2f9ab26aa2f7e9dc0d, State: Running, Role: LEADER
I20260924 09:56:04.056851 3419 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6689502fe43b4a2f9ab26aa2f7e9dc0d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6689502fe43b4a2f9ab26aa2f7e9dc0d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46231 } }
I20260924 09:56:04.057271 3413 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6689502fe43b4a2f9ab26aa2f7e9dc0d [sys.catalog]: configured and running, proceeding with master startup.
I20260924 09:56:04.060297 3421 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6689502fe43b4a2f9ab26aa2f7e9dc0d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6689502fe43b4a2f9ab26aa2f7e9dc0d. Latest consensus state: current_term: 1 leader_uuid: "6689502fe43b4a2f9ab26aa2f7e9dc0d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6689502fe43b4a2f9ab26aa2f7e9dc0d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46231 } } }
I20260924 09:56:04.060746 3421 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6689502fe43b4a2f9ab26aa2f7e9dc0d [sys.catalog]: This master's current role is: LEADER
I20260924 09:56:04.067862 3420 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6689502fe43b4a2f9ab26aa2f7e9dc0d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6689502fe43b4a2f9ab26aa2f7e9dc0d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6689502fe43b4a2f9ab26aa2f7e9dc0d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46231 } } }
I20260924 09:56:04.068187 3420 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6689502fe43b4a2f9ab26aa2f7e9dc0d [sys.catalog]: This master's current role is: LEADER
I20260924 09:56:04.068645 3428 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 09:56:04.073349 3428 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 09:56:04.083940 3438 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260924 09:56:04.087018 3428 catalog_manager.cc:1347] Generated new cluster ID: 4093198082ff4dfd848c18b918d3f928
I20260924 09:56:04.087101 3428 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 09:56:04.101420 3416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:04.086648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43946 (local address 127.0.97.62:46231)
0924 09:56:04.086833 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0924 09:56:04.086839 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 09:56:04.093629 (+ 6790us) server_negotiation.cc:407] Connection header received
0924 09:56:04.093814 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:56:04.093820 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:56:04.093895 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:56:04.093985 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:56:04.095164 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:04.096273 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:04.098038 (+ 1765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:04.098654 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:04.098755 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:04.099837 (+ 1082us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:56:04.099843 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:56:04.099848 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 09:56:04.100594 (+ 746us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:56:04.100639 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 09:56:04.100671 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:56:04.100758 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0924 09:56:04.100903 (+ 145us) server_negotiation.cc:299] Negotiation successful
0924 09:56:04.100990 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":51}
I20260924 09:56:04.114140 3428 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 09:56:04.115309 3428 catalog_manager.cc:1504] Loading token signing keys...
I20260924 09:56:04.123337 3428 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6689502fe43b4a2f9ab26aa2f7e9dc0d: Generated new TSK 0
I20260924 09:56:04.124058 3428 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 09:56:04.134224 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu pid 3346
I20260924 09:56:04.135375 3408 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 09:56:04.223959 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu with pid 3346
2026-09-24T09:56:04Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1733 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790243735634835-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 09:56:04 dist-test-slave-w8d2 krb5kdc[3447](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 09:56:04 dist-test-slave-w8d2 krb5kdc[3447](info): set up 2 sockets
Sep 24 09:56:04 dist-test-slave-w8d2 krb5kdc[3447](info): commencing operation
krb5kdc: starting...
W20260924 09:56:05.207196 388 mini_kdc.cc:121] Time spent starting KDC: real 0.952s user 0.000s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790243735634835-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790243735634835-388-0/krb5kdc/test-user.keytab.
Sep 24 09:56:05 dist-test-slave-w8d2 krb5kdc[3447](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243765, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T09:56:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T09:56:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790243735634835-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790243735634835-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260924 09:56:05.370612 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
/tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790243735634835-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790243735634835-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790243735634835-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790243735634835-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39459
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790243735634835-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37029
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:39459
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790243735634835-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790243735634835-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790243735634835-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 09:56:05.667593 3460 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 09:56:05.667860 3460 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 09:56:05.667968 3460 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 09:56:05.674906 3460 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 09:56:05.675004 3460 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 09:56:05.675073 3460 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 09:56:05.675139 3460 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 09:56:05.675194 3460 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 09:56:05.684648 3460 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37029
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790243735634835-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790243735634835-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:39459
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790243735634835-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39459
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790243735634835-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790243735634835-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 09:56:05.685894 3460 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 09:56:05.687147 3460 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 09:56:05.697952 3465 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:56:05.697939 3468 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:56:05.697908 3466 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 09:56:05.698797 3460 server_base.cc:1034] running on GCE node
I20260924 09:56:05.699543 3460 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 09:56:05.700882 3460 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 09:56:05.702075 3460 hybrid_clock.cc:648] HybridClock initialized: now 1790243765702030 us; error 35 us; skew 500 ppm
I20260924 09:56:05.702390 3460 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 09:56:05 dist-test-slave-w8d2 krb5kdc[3447](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243765, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 09:56:05.705399 3460 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260924 09:56:05.706727 3460 webserver.cc:460] Webserver started at http://127.0.97.62:37491/ using document root <none> and password file <none>
I20260924 09:56:05.707401 3460 fs_manager.cc:359] Metadata directory not provided
I20260924 09:56:05.707508 3460 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 09:56:05.707780 3460 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 09:56:05.710530 3460 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790243735634835-388-0/minicluster-data/master-0/data/instance:
uuid: "ab8d5385b4fb49d9ad9695c927551d95"
format_stamp: "Formatted at 2026-09-24 09:56:05 on dist-test-slave-w8d2"
server_key: "68b008a0397fadd9720cda562ff9a607"
server_key_iv: "40c1c73fb91ae1a42ce6e9022f5f0a85"
server_key_version: "encryptionkey@0"
I20260924 09:56:05.711220 3460 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790243735634835-388-0/minicluster-data/master-0/wal/instance:
uuid: "ab8d5385b4fb49d9ad9695c927551d95"
format_stamp: "Formatted at 2026-09-24 09:56:05 on dist-test-slave-w8d2"
server_key: "68b008a0397fadd9720cda562ff9a607"
server_key_iv: "40c1c73fb91ae1a42ce6e9022f5f0a85"
server_key_version: "encryptionkey@0"
I20260924 09:56:05.715903 3460 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260924 09:56:05.718907 3475 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 09:56:05.720194 3460 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260924 09:56:05.720340 3460 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790243735634835-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790243735634835-388-0/minicluster-data/master-0/wal
uuid: "ab8d5385b4fb49d9ad9695c927551d95"
format_stamp: "Formatted at 2026-09-24 09:56:05 on dist-test-slave-w8d2"
server_key: "68b008a0397fadd9720cda562ff9a607"
server_key_iv: "40c1c73fb91ae1a42ce6e9022f5f0a85"
server_key_version: "encryptionkey@0"
I20260924 09:56:05.720486 3460 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790243735634835-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790243735634835-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790243735634835-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 09:56:05.740366 3460 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 09:56:05.744215 3460 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 09:56:05.744439 3460 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 09:56:05.760053 3460 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:39459
I20260924 09:56:05.760082 3526 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:39459 every 8 connection(s)
I20260924 09:56:05.761222 3460 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790243735634835-388-0/minicluster-data/master-0/data/info.pb
I20260924 09:56:05.764482 3527 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 09:56:05.764925 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu as pid 3460
I20260924 09:56:05.765116 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790243735634835-388-0/minicluster-data/master-0/wal/instance
I20260924 09:56:05.765674 388 external_mini_cluster.cc:1428] Setting key 429a228a135587f35826f07c05d38c2d
I20260924 09:56:05.777184 3527 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ab8d5385b4fb49d9ad9695c927551d95: Bootstrap starting.
Sep 24 09:56:05 dist-test-slave-w8d2 krb5kdc[3447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243765, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 09:56:05.779925 3527 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ab8d5385b4fb49d9ad9695c927551d95: Neither blocks nor log segments found. Creating new log.
I20260924 09:56:05.781138 3527 log.cc:824] T 00000000000000000000000000000000 P ab8d5385b4fb49d9ad9695c927551d95: Log is configured to *not* fsync() on all Append() calls
I20260924 09:56:05.784116 3527 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ab8d5385b4fb49d9ad9695c927551d95: No bootstrap required, opened a new log
I20260924 09:56:05.787127 3530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:05.767516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51850 (local address 127.0.97.62:39459)
0924 09:56:05.768256 (+ 740us) server_negotiation.cc:206] Beginning negotiation
0924 09:56:05.768270 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 09:56:05.768315 (+ 45us) server_negotiation.cc:407] Connection header received
0924 09:56:05.770312 (+ 1997us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:56:05.770338 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:56:05.770811 (+ 473us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:56:05.771217 (+ 406us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:56:05.772325 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:05.774159 (+ 1834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:05.775304 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:05.776272 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:05.776430 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:05.779851 (+ 3421us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:56:05.779863 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:56:05.779880 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:56:05.779931 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:56:05.783225 (+ 3294us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:56:05.783825 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:56:05.783832 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:56:05.783839 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:56:05.784019 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:56:05.784326 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:56:05.784331 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:56:05.784334 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:56:05.784966 (+ 632us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:56:05.785241 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0924 09:56:05.785517 (+ 276us) server_negotiation.cc:299] Negotiation successful
0924 09:56:05.785909 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":515,"thread_start_us":297,"threads_started":1}
I20260924 09:56:05.792454 3527 raft_consensus.cc:348] T 00000000000000000000000000000000 P ab8d5385b4fb49d9ad9695c927551d95 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab8d5385b4fb49d9ad9695c927551d95" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39459 } }
I20260924 09:56:05.792758 3527 raft_consensus.cc:374] T 00000000000000000000000000000000 P ab8d5385b4fb49d9ad9695c927551d95 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 09:56:05.792824 3527 raft_consensus.cc:729] T 00000000000000000000000000000000 P ab8d5385b4fb49d9ad9695c927551d95 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab8d5385b4fb49d9ad9695c927551d95, State: Initialized, Role: FOLLOWER
I20260924 09:56:05.793370 3527 consensus_queue.cc:260] T 00000000000000000000000000000000 P ab8d5385b4fb49d9ad9695c927551d95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab8d5385b4fb49d9ad9695c927551d95" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39459 } }
I20260924 09:56:05.793618 3527 raft_consensus.cc:388] T 00000000000000000000000000000000 P ab8d5385b4fb49d9ad9695c927551d95 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 09:56:05.793751 3527 raft_consensus.cc:482] T 00000000000000000000000000000000 P ab8d5385b4fb49d9ad9695c927551d95 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 09:56:05.793913 3527 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ab8d5385b4fb49d9ad9695c927551d95 [term 0 FOLLOWER]: Advancing to term 1
I20260924 09:56:05.795920 3527 raft_consensus.cc:504] T 00000000000000000000000000000000 P ab8d5385b4fb49d9ad9695c927551d95 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab8d5385b4fb49d9ad9695c927551d95" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39459 } }
I20260924 09:56:05.796339 3527 leader_election.cc:302] T 00000000000000000000000000000000 P ab8d5385b4fb49d9ad9695c927551d95 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ab8d5385b4fb49d9ad9695c927551d95; no voters:
I20260924 09:56:05.796957 3527 leader_election.cc:288] T 00000000000000000000000000000000 P ab8d5385b4fb49d9ad9695c927551d95 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 09:56:05.797061 3532 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ab8d5385b4fb49d9ad9695c927551d95 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 09:56:05.797696 3532 raft_consensus.cc:686] T 00000000000000000000000000000000 P ab8d5385b4fb49d9ad9695c927551d95 [term 1 LEADER]: Becoming Leader. State: Replica: ab8d5385b4fb49d9ad9695c927551d95, State: Running, Role: LEADER
I20260924 09:56:05.798173 3532 consensus_queue.cc:237] T 00000000000000000000000000000000 P ab8d5385b4fb49d9ad9695c927551d95 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab8d5385b4fb49d9ad9695c927551d95" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39459 } }
I20260924 09:56:05.798749 3527 sys_catalog.cc:564] T 00000000000000000000000000000000 P ab8d5385b4fb49d9ad9695c927551d95 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 09:56:05.800925 3533 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab8d5385b4fb49d9ad9695c927551d95 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ab8d5385b4fb49d9ad9695c927551d95" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab8d5385b4fb49d9ad9695c927551d95" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39459 } } }
I20260924 09:56:05.801224 3533 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab8d5385b4fb49d9ad9695c927551d95 [sys.catalog]: This master's current role is: LEADER
I20260924 09:56:05.802356 3534 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab8d5385b4fb49d9ad9695c927551d95 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ab8d5385b4fb49d9ad9695c927551d95. Latest consensus state: current_term: 1 leader_uuid: "ab8d5385b4fb49d9ad9695c927551d95" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab8d5385b4fb49d9ad9695c927551d95" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39459 } } }
I20260924 09:56:05.803514 3534 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab8d5385b4fb49d9ad9695c927551d95 [sys.catalog]: This master's current role is: LEADER
I20260924 09:56:05.804531 3540 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 09:56:05.808704 3540 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 09:56:05.815599 3540 catalog_manager.cc:1347] Generated new cluster ID: 73f3f30f53ce4d6491d2c98b01ab3fdc
I20260924 09:56:05.815696 3540 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 09:56:05.822697 3540 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 09:56:05.823992 3540 catalog_manager.cc:1504] Loading token signing keys...
I20260924 09:56:05.832717 3540 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ab8d5385b4fb49d9ad9695c927551d95: Generated new TSK 0
I20260924 09:56:05.833525 3540 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790243735634835-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790243735634835-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260924 09:56:05.881270 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
/tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790243735634835-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790243735634835-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790243735634835-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790243735634835-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39459
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790243735634835-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37029
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790243735634835-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790243735634835-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790243735634835-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 09:56:06.178785 3553 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 09:56:06.179049 3553 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 09:56:06.179164 3553 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 09:56:06.186041 3553 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 09:56:06.186144 3553 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 09:56:06.186295 3553 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260924 09:56:06.195420 3553 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37029
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790243735634835-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790243735634835-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790243735634835-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790243735634835-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39459
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790243735634835-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 09:56:06.196569 3553 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 09:56:06.197772 3553 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 09:56:06.209355 3558 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:56:06.209414 3559 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:56:06.209800 3561 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 09:56:06.210146 3553 server_base.cc:1034] running on GCE node
I20260924 09:56:06.210671 3553 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 09:56:06.211544 3553 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 09:56:06.212733 3553 hybrid_clock.cc:648] HybridClock initialized: now 1790243766212674 us; error 53 us; skew 500 ppm
I20260924 09:56:06.213006 3553 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 09:56:06 dist-test-slave-w8d2 krb5kdc[3447](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243766, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 09:56:06.215881 3553 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260924 09:56:06.217125 3553 webserver.cc:460] Webserver started at http://127.0.97.1:37307/ using document root <none> and password file <none>
I20260924 09:56:06.217747 3553 fs_manager.cc:359] Metadata directory not provided
I20260924 09:56:06.217852 3553 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 09:56:06.218110 3553 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 09:56:06.220804 3553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790243735634835-388-0/minicluster-data/ts-0/data/instance:
uuid: "31be5fe5b57045dd9834f4336c289eb1"
format_stamp: "Formatted at 2026-09-24 09:56:06 on dist-test-slave-w8d2"
server_key: "e32e138e613e7f00c119f3f50f52ec81"
server_key_iv: "78765ebe4d049b4c9638e835506e2c23"
server_key_version: "encryptionkey@0"
I20260924 09:56:06.221499 3553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790243735634835-388-0/minicluster-data/ts-0/wal/instance:
uuid: "31be5fe5b57045dd9834f4336c289eb1"
format_stamp: "Formatted at 2026-09-24 09:56:06 on dist-test-slave-w8d2"
server_key: "e32e138e613e7f00c119f3f50f52ec81"
server_key_iv: "78765ebe4d049b4c9638e835506e2c23"
server_key_version: "encryptionkey@0"
I20260924 09:56:06.226179 3553 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260924 09:56:06.229173 3568 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 09:56:06.230332 3553 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260924 09:56:06.230470 3553 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790243735634835-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790243735634835-388-0/minicluster-data/ts-0/wal
uuid: "31be5fe5b57045dd9834f4336c289eb1"
format_stamp: "Formatted at 2026-09-24 09:56:06 on dist-test-slave-w8d2"
server_key: "e32e138e613e7f00c119f3f50f52ec81"
server_key_iv: "78765ebe4d049b4c9638e835506e2c23"
server_key_version: "encryptionkey@0"
I20260924 09:56:06.230624 3553 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790243735634835-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790243735634835-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790243735634835-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 09:56:06.241493 3553 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 09:56:06.245235 3553 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 09:56:06.245493 3553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 09:56:06.246309 3553 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 09:56:06.247628 3553 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 09:56:06.247701 3553 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 09:56:06.247813 3553 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 09:56:06.247864 3553 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 09:56:06.275369 3553 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:41075
I20260924 09:56:06.275414 3680 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:41075 every 8 connection(s)
I20260924 09:56:06.276512 3553 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790243735634835-388-0/minicluster-data/ts-0/data/info.pb
I20260924 09:56:06.276901 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu as pid 3553
I20260924 09:56:06.277081 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790243735634835-388-0/minicluster-data/ts-0/wal/instance
I20260924 09:56:06.277945 388 external_mini_cluster.cc:1428] Setting key c90439a44b14552aeb33d9df2578c6ab
Sep 24 09:56:06 dist-test-slave-w8d2 krb5kdc[3447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243766, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260924 09:56:06.295606 3530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:06.279146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:58195 (local address 127.0.97.62:39459)
0924 09:56:06.279358 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0924 09:56:06.279364 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 09:56:06.280490 (+ 1126us) server_negotiation.cc:407] Connection header received
0924 09:56:06.282178 (+ 1688us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:56:06.282183 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:56:06.282256 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:56:06.282336 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:56:06.283762 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:06.284444 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:06.285910 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:06.286288 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:06.286395 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:06.289704 (+ 3309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:56:06.289709 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:56:06.289712 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:56:06.289742 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:56:06.292724 (+ 2982us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:56:06.293400 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:56:06.293405 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:56:06.293409 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:56:06.293545 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:56:06.293990 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:56:06.293995 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:56:06.293999 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:56:06.294362 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:56:06.294468 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0924 09:56:06.295185 (+ 717us) server_negotiation.cc:299] Negotiation successful
0924 09:56:06.295377 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260924 09:56:06.296630 3683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:06.279662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39459 (local address 127.0.97.1:58195)
0924 09:56:06.280353 (+ 691us) negotiation.cc:107] Waiting for socket to connect
0924 09:56:06.280394 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 09:56:06.281937 (+ 1543us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:56:06.282572 (+ 635us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:56:06.282580 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:56:06.283037 (+ 457us) client_negotiation.cc:189] Negotiated authn=SASL
0924 09:56:06.283612 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:06.283629 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:06.284581 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:06.284585 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:06.285783 (+ 1198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:06.285794 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:06.286491 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:06.286498 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:06.286709 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:06.287344 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 09:56:06.287372 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0924 09:56:06.289508 (+ 2136us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 09:56:06.292859 (+ 3351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 09:56:06.292865 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 09:56:06.292877 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0924 09:56:06.293270 (+ 393us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 09:56:06.293668 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 09:56:06.293672 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 09:56:06.293676 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 09:56:06.293872 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 09:56:06.294475 (+ 603us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 09:56:06.294481 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 09:56:06.294794 (+ 313us) client_negotiation.cc:769] Sending connection context
0924 09:56:06.295078 (+ 284us) client_negotiation.cc:240] Negotiation successful
0924 09:56:06.295485 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":288,"threads_started":1}
I20260924 09:56:06.298046 3681 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39459
I20260924 09:56:06.298344 3681 heartbeater.cc:464] Registering TS with master...
I20260924 09:56:06.299047 3681 heartbeater.cc:511] Master 127.0.97.62:39459 requested a full tablet report, sending...
I20260924 09:56:06.300973 3492 ts_manager.cc:194] Registered new tserver with Master: 31be5fe5b57045dd9834f4336c289eb1 (127.0.97.1:41075)
I20260924 09:56:06.302867 3492 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:58195
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790243735634835-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790243735634835-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260924 09:56:06.316042 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
/tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790243735634835-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790243735634835-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790243735634835-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790243735634835-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39459
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790243735634835-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37029
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790243735634835-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790243735634835-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790243735634835-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 09:56:06.613938 3686 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 09:56:06.614178 3686 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 09:56:06.614305 3686 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 09:56:06.621228 3686 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 09:56:06.621333 3686 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 09:56:06.621487 3686 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260924 09:56:06.630686 3686 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37029
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790243735634835-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790243735634835-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790243735634835-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790243735634835-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39459
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790243735634835-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 09:56:06.631906 3686 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 09:56:06.633123 3686 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 09:56:06.645226 3691 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:56:06.645188 3694 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:56:06.645248 3692 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 09:56:06.645957 3686 server_base.cc:1034] running on GCE node
I20260924 09:56:06.646474 3686 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 09:56:06.647308 3686 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 09:56:06.648505 3686 hybrid_clock.cc:648] HybridClock initialized: now 1790243766648458 us; error 41 us; skew 500 ppm
I20260924 09:56:06.648782 3686 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 09:56:06 dist-test-slave-w8d2 krb5kdc[3447](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243766, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 09:56:06.651506 3686 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260924 09:56:06.652745 3686 webserver.cc:460] Webserver started at http://127.0.97.2:45663/ using document root <none> and password file <none>
I20260924 09:56:06.653357 3686 fs_manager.cc:359] Metadata directory not provided
I20260924 09:56:06.653465 3686 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 09:56:06.653718 3686 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 09:56:06.656397 3686 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790243735634835-388-0/minicluster-data/ts-1/data/instance:
uuid: "9e70b02fb7cc43119438074898862e90"
format_stamp: "Formatted at 2026-09-24 09:56:06 on dist-test-slave-w8d2"
server_key: "fb70992f23f72962fab1e6c3ba1b7fb2"
server_key_iv: "767371974b23635c5e0503012a05893e"
server_key_version: "encryptionkey@0"
I20260924 09:56:06.657092 3686 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790243735634835-388-0/minicluster-data/ts-1/wal/instance:
uuid: "9e70b02fb7cc43119438074898862e90"
format_stamp: "Formatted at 2026-09-24 09:56:06 on dist-test-slave-w8d2"
server_key: "fb70992f23f72962fab1e6c3ba1b7fb2"
server_key_iv: "767371974b23635c5e0503012a05893e"
server_key_version: "encryptionkey@0"
I20260924 09:56:06.661545 3686 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 09:56:06.664520 3701 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 09:56:06.665679 3686 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260924 09:56:06.665807 3686 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790243735634835-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790243735634835-388-0/minicluster-data/ts-1/wal
uuid: "9e70b02fb7cc43119438074898862e90"
format_stamp: "Formatted at 2026-09-24 09:56:06 on dist-test-slave-w8d2"
server_key: "fb70992f23f72962fab1e6c3ba1b7fb2"
server_key_iv: "767371974b23635c5e0503012a05893e"
server_key_version: "encryptionkey@0"
I20260924 09:56:06.665980 3686 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790243735634835-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790243735634835-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790243735634835-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 09:56:06.678719 3686 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 09:56:06.681993 3686 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 09:56:06.682236 3686 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 09:56:06.683017 3686 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 09:56:06.684207 3686 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 09:56:06.684280 3686 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 09:56:06.684391 3686 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 09:56:06.684445 3686 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 09:56:06.712227 3686 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:36621
I20260924 09:56:06.712270 3813 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:36621 every 8 connection(s)
I20260924 09:56:06.713358 3686 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790243735634835-388-0/minicluster-data/ts-1/data/info.pb
I20260924 09:56:06.719408 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu as pid 3686
I20260924 09:56:06.719595 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790243735634835-388-0/minicluster-data/ts-1/wal/instance
I20260924 09:56:06.720126 388 external_mini_cluster.cc:1428] Setting key d15ab30509dd0348d09bcce990315598
Sep 24 09:56:06 dist-test-slave-w8d2 krb5kdc[3447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243766, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 09:56:06.731825 3530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:06.715502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:44919 (local address 127.0.97.62:39459)
0924 09:56:06.715700 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0924 09:56:06.715706 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 09:56:06.716811 (+ 1105us) server_negotiation.cc:407] Connection header received
0924 09:56:06.718421 (+ 1610us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:56:06.718425 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:56:06.718489 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:56:06.718571 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:56:06.720243 (+ 1672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:06.721097 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:06.723026 (+ 1929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:06.723471 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:06.723574 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:06.726866 (+ 3292us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:56:06.726871 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:56:06.726874 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:56:06.726903 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:56:06.729117 (+ 2214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:56:06.729733 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:56:06.729738 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:56:06.729741 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:56:06.729852 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:56:06.730221 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:56:06.730226 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:56:06.730229 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:56:06.730475 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:56:06.730606 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0924 09:56:06.731268 (+ 662us) server_negotiation.cc:299] Negotiation successful
0924 09:56:06.731494 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 09:56:06.732662 3816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:06.716036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39459 (local address 127.0.97.2:44919)
0924 09:56:06.716670 (+ 634us) negotiation.cc:107] Waiting for socket to connect
0924 09:56:06.716710 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 09:56:06.718204 (+ 1494us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:56:06.718734 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:56:06.718741 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:56:06.719351 (+ 610us) client_negotiation.cc:189] Negotiated authn=SASL
0924 09:56:06.720078 (+ 727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:06.720103 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:06.721234 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:06.721237 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:06.722889 (+ 1652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:06.722903 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:06.723678 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:06.723686 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:06.723919 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:06.724540 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 09:56:06.724560 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 09:56:06.726681 (+ 2121us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 09:56:06.729244 (+ 2563us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 09:56:06.729252 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 09:56:06.729268 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 09:56:06.729632 (+ 364us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 09:56:06.729954 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 09:56:06.729959 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 09:56:06.729963 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 09:56:06.730126 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 09:56:06.730586 (+ 460us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 09:56:06.730594 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 09:56:06.730884 (+ 290us) client_negotiation.cc:769] Sending connection context
0924 09:56:06.731113 (+ 229us) client_negotiation.cc:240] Negotiation successful
0924 09:56:06.731603 (+ 490us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":250,"threads_started":1}
I20260924 09:56:06.734019 3814 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39459
I20260924 09:56:06.734328 3814 heartbeater.cc:464] Registering TS with master...
I20260924 09:56:06.735015 3814 heartbeater.cc:511] Master 127.0.97.62:39459 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260924 09:56:06.736336 3492 ts_manager.cc:194] Registered new tserver with Master: 9e70b02fb7cc43119438074898862e90 (127.0.97.2:36621)
I20260924 09:56:06.737057 3492 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:44919
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790243735634835-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790243735634835-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260924 09:56:06.759828 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
/tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790243735634835-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790243735634835-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790243735634835-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790243735634835-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39459
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790243735634835-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37029
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790243735634835-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790243735634835-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790243735634835-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 09:56:07.056833 3819 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 09:56:07.057094 3819 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 09:56:07.057191 3819 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 09:56:07.064052 3819 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 09:56:07.064143 3819 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 09:56:07.064254 3819 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260924 09:56:07.073315 3819 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37029
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790243735634835-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790243735634835-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790243735634835-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790243735634835-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39459
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790243735634835-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 09:56:07.074417 3819 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 09:56:07.075623 3819 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 09:56:07.087323 3827 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:56:07.087313 3825 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:56:07.087567 3824 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 09:56:07.088102 3819 server_base.cc:1034] running on GCE node
I20260924 09:56:07.088626 3819 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 09:56:07.089424 3819 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 09:56:07.090607 3819 hybrid_clock.cc:648] HybridClock initialized: now 1790243767090563 us; error 37 us; skew 500 ppm
I20260924 09:56:07.090873 3819 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 09:56:07 dist-test-slave-w8d2 krb5kdc[3447](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243767, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 09:56:07.093588 3819 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260924 09:56:07.094833 3819 webserver.cc:460] Webserver started at http://127.0.97.3:38087/ using document root <none> and password file <none>
I20260924 09:56:07.095638 3819 fs_manager.cc:359] Metadata directory not provided
I20260924 09:56:07.095745 3819 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 09:56:07.096006 3819 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 09:56:07.098582 3819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790243735634835-388-0/minicluster-data/ts-2/data/instance:
uuid: "bbe4392b81964857aebe76142f7a083b"
format_stamp: "Formatted at 2026-09-24 09:56:07 on dist-test-slave-w8d2"
server_key: "35b9fb9a97d8f2d5cd92bdbfa76f7521"
server_key_iv: "06b89ffaa28d44d0061186bef3ba805d"
server_key_version: "encryptionkey@0"
I20260924 09:56:07.099519 3819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790243735634835-388-0/minicluster-data/ts-2/wal/instance:
uuid: "bbe4392b81964857aebe76142f7a083b"
format_stamp: "Formatted at 2026-09-24 09:56:07 on dist-test-slave-w8d2"
server_key: "35b9fb9a97d8f2d5cd92bdbfa76f7521"
server_key_iv: "06b89ffaa28d44d0061186bef3ba805d"
server_key_version: "encryptionkey@0"
I20260924 09:56:07.104741 3819 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260924 09:56:07.107837 3834 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 09:56:07.109033 3819 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 09:56:07.109174 3819 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790243735634835-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790243735634835-388-0/minicluster-data/ts-2/wal
uuid: "bbe4392b81964857aebe76142f7a083b"
format_stamp: "Formatted at 2026-09-24 09:56:07 on dist-test-slave-w8d2"
server_key: "35b9fb9a97d8f2d5cd92bdbfa76f7521"
server_key_iv: "06b89ffaa28d44d0061186bef3ba805d"
server_key_version: "encryptionkey@0"
I20260924 09:56:07.109332 3819 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790243735634835-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790243735634835-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790243735634835-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 09:56:07.150620 3819 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 09:56:07.154274 3819 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 09:56:07.154533 3819 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 09:56:07.155450 3819 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 09:56:07.156656 3819 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 09:56:07.156723 3819 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 09:56:07.156828 3819 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 09:56:07.156874 3819 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 09:56:07.184857 3819 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:38233
I20260924 09:56:07.184906 3946 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:38233 every 8 connection(s)
I20260924 09:56:07.186002 3819 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790243735634835-388-0/minicluster-data/ts-2/data/info.pb
I20260924 09:56:07.195519 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu as pid 3819
I20260924 09:56:07.195720 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790243735634835-388-0/minicluster-data/ts-2/wal/instance
I20260924 09:56:07.196257 388 external_mini_cluster.cc:1428] Setting key 1f93d1b0bdf2d8ffe7b897958d455f0b
Sep 24 09:56:07 dist-test-slave-w8d2 krb5kdc[3447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243767, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 09:56:07.203315 3530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:07.188219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:52279 (local address 127.0.97.62:39459)
0924 09:56:07.188363 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0924 09:56:07.188367 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 09:56:07.189441 (+ 1074us) server_negotiation.cc:407] Connection header received
0924 09:56:07.191101 (+ 1660us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:56:07.191106 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:56:07.191180 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:56:07.191285 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:56:07.192620 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:07.193288 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:07.194737 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:07.195125 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:07.195250 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:07.198683 (+ 3433us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:56:07.198689 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:56:07.198694 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:56:07.198729 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:56:07.201070 (+ 2341us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:56:07.201653 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:56:07.201657 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:56:07.201659 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:56:07.201761 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:56:07.202058 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:56:07.202061 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:56:07.202063 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:56:07.202305 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:56:07.202413 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0924 09:56:07.202918 (+ 505us) server_negotiation.cc:299] Negotiation successful
0924 09:56:07.203068 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 09:56:07.204315 3949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:07.188638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39459 (local address 127.0.97.3:52279)
0924 09:56:07.189287 (+ 649us) negotiation.cc:107] Waiting for socket to connect
0924 09:56:07.189323 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 09:56:07.190849 (+ 1526us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:56:07.191492 (+ 643us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:56:07.191500 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:56:07.191954 (+ 454us) client_negotiation.cc:189] Negotiated authn=SASL
0924 09:56:07.192460 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:07.192476 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:07.193426 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:07.193431 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:07.194610 (+ 1179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:07.194621 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:07.195348 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:07.195355 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:07.195560 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:07.196236 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 09:56:07.196255 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 09:56:07.198487 (+ 2232us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 09:56:07.201178 (+ 2691us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 09:56:07.201184 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 09:56:07.201196 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0924 09:56:07.201565 (+ 369us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 09:56:07.201829 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 09:56:07.201832 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 09:56:07.201834 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 09:56:07.201988 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 09:56:07.202389 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 09:56:07.202395 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 09:56:07.202664 (+ 269us) client_negotiation.cc:769] Sending connection context
0924 09:56:07.202869 (+ 205us) client_negotiation.cc:240] Negotiation successful
0924 09:56:07.203221 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":269,"threads_started":1}
I20260924 09:56:07.205446 3947 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39459
I20260924 09:56:07.205724 3947 heartbeater.cc:464] Registering TS with master...
I20260924 09:56:07.206367 3947 heartbeater.cc:511] Master 127.0.97.62:39459 requested a full tablet report, sending...
I20260924 09:56:07.207592 3492 ts_manager.cc:194] Registered new tserver with Master: bbe4392b81964857aebe76142f7a083b (127.0.97.3:38233)
I20260924 09:56:07.208307 3492 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:52279
I20260924 09:56:07.210825 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 09:56:07 dist-test-slave-w8d2 krb5kdc[3447](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243767, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 24 09:56:07 dist-test-slave-w8d2 krb5kdc[3447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243767, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 09:56:07.251006 3530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:07.238406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51860 (local address 127.0.97.62:39459)
0924 09:56:07.238563 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0924 09:56:07.238568 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 09:56:07.238942 (+ 374us) server_negotiation.cc:407] Connection header received
0924 09:56:07.239104 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:56:07.239110 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:56:07.239169 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:56:07.239285 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:56:07.240563 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:07.241307 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:07.242471 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:07.242846 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:07.242928 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:07.246421 (+ 3493us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:56:07.246426 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:56:07.246429 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:56:07.246469 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:56:07.248691 (+ 2222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:56:07.249259 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:56:07.249264 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:56:07.249266 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:56:07.249369 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:56:07.249715 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:56:07.249720 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:56:07.249723 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:56:07.250011 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:56:07.250246 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0924 09:56:07.250492 (+ 246us) server_negotiation.cc:299] Negotiation successful
0924 09:56:07.250694 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
W20260924 09:56:07.251554 3492 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:51860
I20260924 09:56:07.252780 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu pid 3553
Sep 24 09:56:07 dist-test-slave-w8d2 krb5kdc[3447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243767, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260924 09:56:07.267216 3958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:07.254075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57404 (local address 127.0.97.1:41075)
0924 09:56:07.254592 (+ 517us) server_negotiation.cc:206] Beginning negotiation
0924 09:56:07.254599 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 09:56:07.254621 (+ 22us) server_negotiation.cc:407] Connection header received
0924 09:56:07.254671 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:56:07.254677 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:56:07.254872 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:56:07.255008 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:56:07.256111 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:07.257112 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:07.258266 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:07.258805 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:07.258913 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:07.262373 (+ 3460us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:56:07.262383 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:56:07.262397 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:56:07.262426 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:56:07.264778 (+ 2352us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:56:07.265371 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:56:07.265379 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:56:07.265381 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:56:07.265495 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:56:07.265827 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:56:07.265832 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:56:07.265834 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:56:07.266132 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:56:07.266401 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0924 09:56:07.266669 (+ 268us) server_negotiation.cc:299] Negotiation successful
0924 09:56:07.266907 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":388,"thread_start_us":253,"threads_started":1}
W20260924 09:56:07.268368 3676 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:57404
W20260924 09:56:07.269435 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu pid 3553: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 09:56:07.269587 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu with pid 3553
I20260924 09:56:07.281577 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu pid 3686
Sep 24 09:56:07 dist-test-slave-w8d2 krb5kdc[3447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243767, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260924 09:56:07.295468 3959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:07.282410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39368 (local address 127.0.97.2:36621)
0924 09:56:07.282922 (+ 512us) server_negotiation.cc:206] Beginning negotiation
0924 09:56:07.282928 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 09:56:07.282953 (+ 25us) server_negotiation.cc:407] Connection header received
0924 09:56:07.283015 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:56:07.283021 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:56:07.283221 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:56:07.283365 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:56:07.284417 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:07.285390 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:07.286542 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:07.287052 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:07.287143 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:07.290760 (+ 3617us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:56:07.290770 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:56:07.290783 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:56:07.290812 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:56:07.293205 (+ 2393us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:56:07.293760 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:56:07.293765 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:56:07.293768 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:56:07.293870 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:56:07.294192 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:56:07.294195 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:56:07.294197 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:56:07.294473 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:56:07.294680 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0924 09:56:07.294903 (+ 223us) server_negotiation.cc:299] Negotiation successful
0924 09:56:07.295083 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":384,"thread_start_us":286,"threads_started":1}
W20260924 09:56:07.296677 3809 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:39368
W20260924 09:56:07.297638 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu pid 3686: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 09:56:07.297771 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu with pid 3686
I20260924 09:56:07.309494 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu pid 3819
Sep 24 09:56:07 dist-test-slave-w8d2 krb5kdc[3447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243767, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260924 09:56:07.324299 3960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:07.310340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46664 (local address 127.0.97.3:38233)
0924 09:56:07.310882 (+ 542us) server_negotiation.cc:206] Beginning negotiation
0924 09:56:07.310889 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 09:56:07.310916 (+ 27us) server_negotiation.cc:407] Connection header received
0924 09:56:07.310971 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:56:07.310978 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:56:07.311174 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:56:07.311355 (+ 181us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:56:07.312660 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:07.313707 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:07.314889 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:07.315524 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:07.315639 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:07.319510 (+ 3871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:56:07.319521 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:56:07.319535 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:56:07.319565 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:56:07.321926 (+ 2361us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:56:07.322521 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:56:07.322527 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:56:07.322529 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:56:07.322638 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:56:07.322955 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:56:07.322959 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:56:07.322963 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:56:07.323306 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:56:07.323554 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0924 09:56:07.323748 (+ 194us) server_negotiation.cc:299] Negotiation successful
0924 09:56:07.323957 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":414,"thread_start_us":278,"threads_started":1}
W20260924 09:56:07.325476 3942 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:46664
W20260924 09:56:07.326483 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu pid 3819: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 09:56:07.326611 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu with pid 3819
I20260924 09:56:07.338240 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu pid 3460
I20260924 09:56:07.339398 3522 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 09:56:07.437551 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu with pid 3460
2026-09-24T09:56:07Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3218 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790243735634835-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 09:56:07 dist-test-slave-w8d2 krb5kdc[3964](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 09:56:07 dist-test-slave-w8d2 krb5kdc[3964](info): set up 2 sockets
Sep 24 09:56:07 dist-test-slave-w8d2 krb5kdc[3964](info): commencing operation
krb5kdc: starting...
W20260924 09:56:08.415287 388 mini_kdc.cc:121] Time spent starting KDC: real 0.942s user 0.000s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790243735634835-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790243735634835-388-0/krb5kdc/test-user.keytab.
Sep 24 09:56:08 dist-test-slave-w8d2 krb5kdc[3964](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243768, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T09:56:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T09:56:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790243735634835-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790243735634835-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260924 09:56:08.581918 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
/tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790243735634835-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790243735634835-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790243735634835-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790243735634835-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44591
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790243735634835-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:44591
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790243735634835-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790243735634835-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790243735634835-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 09:56:08.875566 3977 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 09:56:08.875823 3977 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 09:56:08.875945 3977 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 09:56:08.882711 3977 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 09:56:08.882810 3977 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 09:56:08.882879 3977 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 09:56:08.882954 3977 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 09:56:08.883009 3977 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 09:56:08.892170 3977 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790243735634835-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790243735634835-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:44591
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790243735634835-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44591
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790243735634835-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790243735634835-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 09:56:08.893368 3977 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 09:56:08.894582 3977 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 09:56:08.905812 3982 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:56:08.905892 3983 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:56:08.905862 3985 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 09:56:08.906615 3977 server_base.cc:1034] running on GCE node
I20260924 09:56:08.907392 3977 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 09:56:08.908669 3977 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 09:56:08.909873 3977 hybrid_clock.cc:648] HybridClock initialized: now 1790243768909825 us; error 37 us; skew 500 ppm
I20260924 09:56:08.910181 3977 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 09:56:08 dist-test-slave-w8d2 krb5kdc[3964](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243768, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 09:56:08.912909 3977 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260924 09:56:08.914278 3977 webserver.cc:460] Webserver started at http://127.0.97.62:34887/ using document root <none> and password file <none>
I20260924 09:56:08.914875 3977 fs_manager.cc:359] Metadata directory not provided
I20260924 09:56:08.914983 3977 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 09:56:08.915274 3977 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 09:56:08.917966 3977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790243735634835-388-0/minicluster-data/master-0/data/instance:
uuid: "6fff2533178a48628527884a5497d4bb"
format_stamp: "Formatted at 2026-09-24 09:56:08 on dist-test-slave-w8d2"
server_key: "c01d1a98a000702780ca583a349fbc65"
server_key_iv: "d2f3e69c2e81ae8c4e18208c4f2e0787"
server_key_version: "encryptionkey@0"
I20260924 09:56:08.918658 3977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790243735634835-388-0/minicluster-data/master-0/wal/instance:
uuid: "6fff2533178a48628527884a5497d4bb"
format_stamp: "Formatted at 2026-09-24 09:56:08 on dist-test-slave-w8d2"
server_key: "c01d1a98a000702780ca583a349fbc65"
server_key_iv: "d2f3e69c2e81ae8c4e18208c4f2e0787"
server_key_version: "encryptionkey@0"
I20260924 09:56:08.923188 3977 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260924 09:56:08.926222 3992 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 09:56:08.927403 3977 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 09:56:08.927542 3977 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790243735634835-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790243735634835-388-0/minicluster-data/master-0/wal
uuid: "6fff2533178a48628527884a5497d4bb"
format_stamp: "Formatted at 2026-09-24 09:56:08 on dist-test-slave-w8d2"
server_key: "c01d1a98a000702780ca583a349fbc65"
server_key_iv: "d2f3e69c2e81ae8c4e18208c4f2e0787"
server_key_version: "encryptionkey@0"
I20260924 09:56:08.927697 3977 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790243735634835-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790243735634835-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790243735634835-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 09:56:08.943288 3977 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 09:56:08.946764 3977 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 09:56:08.947006 3977 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 09:56:08.962972 3977 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:44591
I20260924 09:56:08.963002 4043 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:44591 every 8 connection(s)
I20260924 09:56:08.964342 3977 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790243735634835-388-0/minicluster-data/master-0/data/info.pb
I20260924 09:56:08.965709 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu as pid 3977
I20260924 09:56:08.965883 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790243735634835-388-0/minicluster-data/master-0/wal/instance
I20260924 09:56:08.966451 388 external_mini_cluster.cc:1428] Setting key ea3730b28a2a5a0daae072101eb5964f
I20260924 09:56:08.968178 4044 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 24 09:56:08 dist-test-slave-w8d2 krb5kdc[3964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243768, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 09:56:08.980839 4044 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6fff2533178a48628527884a5497d4bb: Bootstrap starting.
I20260924 09:56:08.983860 4044 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6fff2533178a48628527884a5497d4bb: Neither blocks nor log segments found. Creating new log.
I20260924 09:56:08.985036 4044 log.cc:824] T 00000000000000000000000000000000 P 6fff2533178a48628527884a5497d4bb: Log is configured to *not* fsync() on all Append() calls
I20260924 09:56:08.987934 4044 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6fff2533178a48628527884a5497d4bb: No bootstrap required, opened a new log
I20260924 09:56:08.988149 4047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:08.968231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43674 (local address 127.0.97.62:44591)
0924 09:56:08.968949 (+ 718us) server_negotiation.cc:206] Beginning negotiation
0924 09:56:08.968960 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 09:56:08.968999 (+ 39us) server_negotiation.cc:407] Connection header received
0924 09:56:08.970500 (+ 1501us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:56:08.970520 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:56:08.970884 (+ 364us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:56:08.971199 (+ 315us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:56:08.972998 (+ 1799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:08.974300 (+ 1302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:08.975957 (+ 1657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:08.976575 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:08.976695 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:08.981090 (+ 4395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:56:08.981098 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:56:08.981111 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:56:08.981149 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:56:08.983829 (+ 2680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:56:08.984553 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:56:08.984559 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:56:08.984564 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:56:08.984713 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:56:08.985084 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:56:08.985087 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:56:08.985089 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:56:08.985640 (+ 551us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:56:08.985931 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0924 09:56:08.986681 (+ 750us) server_negotiation.cc:299] Negotiation successful
0924 09:56:08.987169 (+ 488us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":521,"thread_start_us":318,"threads_started":1}
I20260924 09:56:08.995687 4044 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6fff2533178a48628527884a5497d4bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fff2533178a48628527884a5497d4bb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44591 } }
I20260924 09:56:08.995991 4044 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6fff2533178a48628527884a5497d4bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 09:56:08.996080 4044 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6fff2533178a48628527884a5497d4bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6fff2533178a48628527884a5497d4bb, State: Initialized, Role: FOLLOWER
I20260924 09:56:08.996630 4044 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6fff2533178a48628527884a5497d4bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fff2533178a48628527884a5497d4bb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44591 } }
I20260924 09:56:08.996884 4044 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6fff2533178a48628527884a5497d4bb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 09:56:08.997021 4044 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6fff2533178a48628527884a5497d4bb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 09:56:08.997185 4044 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6fff2533178a48628527884a5497d4bb [term 0 FOLLOWER]: Advancing to term 1
I20260924 09:56:08.999094 4044 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6fff2533178a48628527884a5497d4bb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fff2533178a48628527884a5497d4bb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44591 } }
I20260924 09:56:08.999543 4044 leader_election.cc:302] T 00000000000000000000000000000000 P 6fff2533178a48628527884a5497d4bb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6fff2533178a48628527884a5497d4bb; no voters:
I20260924 09:56:09.000030 4044 leader_election.cc:288] T 00000000000000000000000000000000 P 6fff2533178a48628527884a5497d4bb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 09:56:09.000140 4049 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6fff2533178a48628527884a5497d4bb [term 1 FOLLOWER]: Leader election won for term 1
I20260924 09:56:09.000695 4049 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6fff2533178a48628527884a5497d4bb [term 1 LEADER]: Becoming Leader. State: Replica: 6fff2533178a48628527884a5497d4bb, State: Running, Role: LEADER
I20260924 09:56:09.001132 4049 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6fff2533178a48628527884a5497d4bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fff2533178a48628527884a5497d4bb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44591 } }
I20260924 09:56:09.001871 4044 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6fff2533178a48628527884a5497d4bb [sys.catalog]: configured and running, proceeding with master startup.
I20260924 09:56:09.003741 4050 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6fff2533178a48628527884a5497d4bb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6fff2533178a48628527884a5497d4bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fff2533178a48628527884a5497d4bb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44591 } } }
I20260924 09:56:09.004112 4050 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6fff2533178a48628527884a5497d4bb [sys.catalog]: This master's current role is: LEADER
I20260924 09:56:09.003928 4051 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6fff2533178a48628527884a5497d4bb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6fff2533178a48628527884a5497d4bb. Latest consensus state: current_term: 1 leader_uuid: "6fff2533178a48628527884a5497d4bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fff2533178a48628527884a5497d4bb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44591 } } }
I20260924 09:56:09.004510 4051 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6fff2533178a48628527884a5497d4bb [sys.catalog]: This master's current role is: LEADER
I20260924 09:56:09.005935 4057 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 09:56:09.010355 4057 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 09:56:09.017967 4057 catalog_manager.cc:1347] Generated new cluster ID: c3d9d274ece24ee0a5f6270c2c97bd00
I20260924 09:56:09.018064 4057 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 09:56:09.033026 4057 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 09:56:09.034224 4057 catalog_manager.cc:1504] Loading token signing keys...
I20260924 09:56:09.041785 4057 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6fff2533178a48628527884a5497d4bb: Generated new TSK 0
I20260924 09:56:09.042512 4057 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790243735634835-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790243735634835-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260924 09:56:09.090741 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
/tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790243735634835-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790243735634835-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790243735634835-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790243735634835-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44591
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790243735634835-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790243735634835-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790243735634835-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790243735634835-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 09:56:09.415834 4070 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 09:56:09.416095 4070 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 09:56:09.416234 4070 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 09:56:09.423193 4070 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 09:56:09.423331 4070 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 09:56:09.423472 4070 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260924 09:56:09.432804 4070 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790243735634835-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790243735634835-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790243735634835-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790243735634835-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44591
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790243735634835-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 09:56:09.433930 4070 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 09:56:09.435128 4070 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 09:56:09.446900 4076 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:56:09.446844 4075 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:56:09.446921 4078 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 09:56:09.447579 4070 server_base.cc:1034] running on GCE node
I20260924 09:56:09.448086 4070 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 09:56:09.448890 4070 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 09:56:09.450083 4070 hybrid_clock.cc:648] HybridClock initialized: now 1790243769450031 us; error 41 us; skew 500 ppm
I20260924 09:56:09.450390 4070 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 09:56:09 dist-test-slave-w8d2 krb5kdc[3964](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243769, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 09:56:09.453150 4070 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260924 09:56:09.454408 4070 webserver.cc:460] Webserver started at http://127.0.97.1:40401/ using document root <none> and password file <none>
I20260924 09:56:09.454989 4070 fs_manager.cc:359] Metadata directory not provided
I20260924 09:56:09.455081 4070 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 09:56:09.455354 4070 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 09:56:09.457980 4070 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790243735634835-388-0/minicluster-data/ts-0/data/instance:
uuid: "6c19d6f67f784941985cdc983ff4d1cd"
format_stamp: "Formatted at 2026-09-24 09:56:09 on dist-test-slave-w8d2"
server_key: "381cd66236cf02c7c7b9f5d1e93cbafd"
server_key_iv: "386e39b6b0fa9c6a8406a44d3ee67fa2"
server_key_version: "encryptionkey@0"
I20260924 09:56:09.458658 4070 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790243735634835-388-0/minicluster-data/ts-0/wal/instance:
uuid: "6c19d6f67f784941985cdc983ff4d1cd"
format_stamp: "Formatted at 2026-09-24 09:56:09 on dist-test-slave-w8d2"
server_key: "381cd66236cf02c7c7b9f5d1e93cbafd"
server_key_iv: "386e39b6b0fa9c6a8406a44d3ee67fa2"
server_key_version: "encryptionkey@0"
I20260924 09:56:09.463158 4070 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260924 09:56:09.466218 4085 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 09:56:09.467496 4070 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 09:56:09.467640 4070 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790243735634835-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790243735634835-388-0/minicluster-data/ts-0/wal
uuid: "6c19d6f67f784941985cdc983ff4d1cd"
format_stamp: "Formatted at 2026-09-24 09:56:09 on dist-test-slave-w8d2"
server_key: "381cd66236cf02c7c7b9f5d1e93cbafd"
server_key_iv: "386e39b6b0fa9c6a8406a44d3ee67fa2"
server_key_version: "encryptionkey@0"
I20260924 09:56:09.467829 4070 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790243735634835-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790243735634835-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790243735634835-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 09:56:09.480901 4070 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 09:56:09.484265 4070 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 09:56:09.484483 4070 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 09:56:09.485242 4070 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 09:56:09.486383 4070 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 09:56:09.486449 4070 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 09:56:09.486531 4070 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 09:56:09.486573 4070 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 09:56:09.515798 4070 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:33419
I20260924 09:56:09.515844 4197 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:33419 every 8 connection(s)
I20260924 09:56:09.516950 4070 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790243735634835-388-0/minicluster-data/ts-0/data/info.pb
I20260924 09:56:09.517392 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu as pid 4070
I20260924 09:56:09.517563 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790243735634835-388-0/minicluster-data/ts-0/wal/instance
I20260924 09:56:09.518313 388 external_mini_cluster.cc:1428] Setting key 1236fc481ce528eded93dffbc31690d7
Sep 24 09:56:09 dist-test-slave-w8d2 krb5kdc[3964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243769, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260924 09:56:09.537132 4200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:09.519322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:57155 (local address 127.0.97.62:44591)
0924 09:56:09.519752 (+ 430us) server_negotiation.cc:206] Beginning negotiation
0924 09:56:09.519758 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 09:56:09.520887 (+ 1129us) server_negotiation.cc:407] Connection header received
0924 09:56:09.523173 (+ 2286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:56:09.523176 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:56:09.523255 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:56:09.523351 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:56:09.525102 (+ 1751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:09.525747 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:09.527197 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:09.527827 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:09.527939 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:09.531157 (+ 3218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:56:09.531163 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:56:09.531168 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:56:09.531201 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:56:09.534223 (+ 3022us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:56:09.534861 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:56:09.534866 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:56:09.534869 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:56:09.535002 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:56:09.535518 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:56:09.535523 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:56:09.535527 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:56:09.535878 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:56:09.536015 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0924 09:56:09.536706 (+ 691us) server_negotiation.cc:299] Negotiation successful
0924 09:56:09.536865 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":213,"threads_started":1}
I20260924 09:56:09.538069 4201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:09.519911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44591 (local address 127.0.97.1:57155)
0924 09:56:09.520721 (+ 810us) negotiation.cc:107] Waiting for socket to connect
0924 09:56:09.520767 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0924 09:56:09.522920 (+ 2153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:56:09.523587 (+ 667us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:56:09.523597 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:56:09.524246 (+ 649us) client_negotiation.cc:189] Negotiated authn=SASL
0924 09:56:09.524962 (+ 716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:09.524985 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:09.525887 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:09.525892 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:09.527069 (+ 1177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:09.527079 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:09.528005 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:09.528010 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:09.528186 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:09.528781 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 09:56:09.528802 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 09:56:09.530935 (+ 2133us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 09:56:09.534355 (+ 3420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 09:56:09.534361 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 09:56:09.534373 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0924 09:56:09.534742 (+ 369us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 09:56:09.535156 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 09:56:09.535159 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 09:56:09.535162 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 09:56:09.535410 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 09:56:09.536022 (+ 612us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 09:56:09.536028 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 09:56:09.536394 (+ 366us) client_negotiation.cc:769] Sending connection context
0924 09:56:09.536623 (+ 229us) client_negotiation.cc:240] Negotiation successful
0924 09:56:09.536998 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":531,"thread_start_us":305,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260924 09:56:09.539579 4198 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44591
I20260924 09:56:09.539896 4198 heartbeater.cc:464] Registering TS with master...
I20260924 09:56:09.540587 4198 heartbeater.cc:511] Master 127.0.97.62:44591 requested a full tablet report, sending...
I20260924 09:56:09.542630 4009 ts_manager.cc:194] Registered new tserver with Master: 6c19d6f67f784941985cdc983ff4d1cd (127.0.97.1:33419)
I20260924 09:56:09.544368 4009 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:57155
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790243735634835-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790243735634835-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260924 09:56:09.560117 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
/tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790243735634835-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790243735634835-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790243735634835-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790243735634835-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44591
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790243735634835-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790243735634835-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790243735634835-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790243735634835-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 09:56:09.859328 4204 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 09:56:09.859583 4204 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 09:56:09.859680 4204 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 09:56:09.866492 4204 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 09:56:09.866585 4204 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 09:56:09.866700 4204 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260924 09:56:09.875938 4204 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790243735634835-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790243735634835-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790243735634835-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790243735634835-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44591
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790243735634835-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 09:56:09.877059 4204 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 09:56:09.878240 4204 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 09:56:09.889910 4209 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:56:09.889995 4210 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:56:09.890022 4212 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 09:56:09.890761 4204 server_base.cc:1034] running on GCE node
I20260924 09:56:09.891311 4204 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 09:56:09.892112 4204 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 09:56:09.893286 4204 hybrid_clock.cc:648] HybridClock initialized: now 1790243769893238 us; error 45 us; skew 500 ppm
I20260924 09:56:09.893534 4204 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 09:56:09 dist-test-slave-w8d2 krb5kdc[3964](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243769, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 09:56:09.896349 4204 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260924 09:56:09.897625 4204 webserver.cc:460] Webserver started at http://127.0.97.2:43687/ using document root <none> and password file <none>
I20260924 09:56:09.898216 4204 fs_manager.cc:359] Metadata directory not provided
I20260924 09:56:09.898330 4204 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 09:56:09.898566 4204 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 09:56:09.901320 4204 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790243735634835-388-0/minicluster-data/ts-1/data/instance:
uuid: "513b7e5215ba47c2b79d0087e3b64888"
format_stamp: "Formatted at 2026-09-24 09:56:09 on dist-test-slave-w8d2"
server_key: "71074e5129a82e8666bd84dd5623607c"
server_key_iv: "09aa44b44a7189e90c65c790ede4e479"
server_key_version: "encryptionkey@0"
I20260924 09:56:09.902050 4204 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790243735634835-388-0/minicluster-data/ts-1/wal/instance:
uuid: "513b7e5215ba47c2b79d0087e3b64888"
format_stamp: "Formatted at 2026-09-24 09:56:09 on dist-test-slave-w8d2"
server_key: "71074e5129a82e8666bd84dd5623607c"
server_key_iv: "09aa44b44a7189e90c65c790ede4e479"
server_key_version: "encryptionkey@0"
I20260924 09:56:09.906742 4204 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260924 09:56:09.909825 4219 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 09:56:09.910997 4204 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 09:56:09.911119 4204 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790243735634835-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790243735634835-388-0/minicluster-data/ts-1/wal
uuid: "513b7e5215ba47c2b79d0087e3b64888"
format_stamp: "Formatted at 2026-09-24 09:56:09 on dist-test-slave-w8d2"
server_key: "71074e5129a82e8666bd84dd5623607c"
server_key_iv: "09aa44b44a7189e90c65c790ede4e479"
server_key_version: "encryptionkey@0"
I20260924 09:56:09.911326 4204 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790243735634835-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790243735634835-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790243735634835-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 09:56:09.925940 4204 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 09:56:09.929708 4204 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 09:56:09.929948 4204 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 09:56:09.930789 4204 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 09:56:09.932041 4204 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 09:56:09.932107 4204 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 09:56:09.932215 4204 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 09:56:09.932271 4204 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 09:56:09.960635 4204 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:34707
I20260924 09:56:09.960680 4331 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:34707 every 8 connection(s)
I20260924 09:56:09.961833 4204 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790243735634835-388-0/minicluster-data/ts-1/data/info.pb
I20260924 09:56:09.963932 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu as pid 4204
I20260924 09:56:09.964084 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790243735634835-388-0/minicluster-data/ts-1/wal/instance
I20260924 09:56:09.964498 388 external_mini_cluster.cc:1428] Setting key 5b2d647b038204ac4c97aef77c094a56
Sep 24 09:56:09 dist-test-slave-w8d2 krb5kdc[3964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243769, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 09:56:09.982319 4200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:09.964876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:41917 (local address 127.0.97.62:44591)
0924 09:56:09.965068 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0924 09:56:09.965074 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 09:56:09.966041 (+ 967us) server_negotiation.cc:407] Connection header received
0924 09:56:09.968509 (+ 2468us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:56:09.968515 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:56:09.968606 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:56:09.968718 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:56:09.970492 (+ 1774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:09.971629 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:09.973364 (+ 1735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:09.973803 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:09.973908 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:09.977025 (+ 3117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:56:09.977030 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:56:09.977033 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:56:09.977062 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:56:09.979618 (+ 2556us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:56:09.980297 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:56:09.980303 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:56:09.980306 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:56:09.980433 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:56:09.980806 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:56:09.980811 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:56:09.980815 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:56:09.981062 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:56:09.981181 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0924 09:56:09.981800 (+ 619us) server_negotiation.cc:299] Negotiation successful
0924 09:56:09.982011 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260924 09:56:09.983225 4334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:09.965080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44591 (local address 127.0.97.2:41917)
0924 09:56:09.965880 (+ 800us) negotiation.cc:107] Waiting for socket to connect
0924 09:56:09.965941 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0924 09:56:09.968210 (+ 2269us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:56:09.968953 (+ 743us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:56:09.968964 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:56:09.969584 (+ 620us) client_negotiation.cc:189] Negotiated authn=SASL
0924 09:56:09.970312 (+ 728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:09.970337 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:09.971794 (+ 1457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:09.971807 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:09.973225 (+ 1418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:09.973236 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:09.974011 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:09.974018 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:09.974225 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:09.974814 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 09:56:09.974835 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 09:56:09.976840 (+ 2005us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 09:56:09.979732 (+ 2892us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 09:56:09.979740 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 09:56:09.979756 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 09:56:09.980196 (+ 440us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 09:56:09.980529 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 09:56:09.980534 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 09:56:09.980538 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 09:56:09.980711 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 09:56:09.981172 (+ 461us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 09:56:09.981179 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 09:56:09.981481 (+ 302us) client_negotiation.cc:769] Sending connection context
0924 09:56:09.981711 (+ 230us) client_negotiation.cc:240] Negotiation successful
0924 09:56:09.982179 (+ 468us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":504,"thread_start_us":517,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260924 09:56:09.984475 4332 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44591
I20260924 09:56:09.984752 4332 heartbeater.cc:464] Registering TS with master...
I20260924 09:56:09.985462 4332 heartbeater.cc:511] Master 127.0.97.62:44591 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260924 09:56:09.986797 4009 ts_manager.cc:194] Registered new tserver with Master: 513b7e5215ba47c2b79d0087e3b64888 (127.0.97.2:34707)
I20260924 09:56:09.987834 4009 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:41917
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790243735634835-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790243735634835-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260924 09:56:10.005604 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
/tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790243735634835-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790243735634835-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790243735634835-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790243735634835-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44591
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790243735634835-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790243735634835-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790243735634835-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790243735634835-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 09:56:10.301512 4337 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 09:56:10.301786 4337 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 09:56:10.301921 4337 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 09:56:10.309015 4337 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 09:56:10.309130 4337 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 09:56:10.309285 4337 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260924 09:56:10.318863 4337 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790243735634835-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790243735634835-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790243735634835-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790243735634835-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44591
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790243735634835-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 09:56:10.320130 4337 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 09:56:10.321425 4337 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 09:56:10.333487 4342 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:56:10.333552 4345 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:56:10.333516 4343 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 09:56:10.334271 4337 server_base.cc:1034] running on GCE node
I20260924 09:56:10.334820 4337 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 09:56:10.335703 4337 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 09:56:10.336889 4337 hybrid_clock.cc:648] HybridClock initialized: now 1790243770336849 us; error 35 us; skew 500 ppm
I20260924 09:56:10.337155 4337 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 09:56:10 dist-test-slave-w8d2 krb5kdc[3964](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243770, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 09:56:10.339996 4337 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260924 09:56:10.341297 4337 webserver.cc:460] Webserver started at http://127.0.97.3:37519/ using document root <none> and password file <none>
I20260924 09:56:10.341877 4337 fs_manager.cc:359] Metadata directory not provided
I20260924 09:56:10.341975 4337 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 09:56:10.342212 4337 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 09:56:10.344919 4337 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790243735634835-388-0/minicluster-data/ts-2/data/instance:
uuid: "f580ddf34a8a4d58ac1f65e7b4ed78e3"
format_stamp: "Formatted at 2026-09-24 09:56:10 on dist-test-slave-w8d2"
server_key: "6e9db5fa5d8a6c0818fc4490c4ec93f0"
server_key_iv: "228288e6d129a683e1d3b33a1eca3595"
server_key_version: "encryptionkey@0"
I20260924 09:56:10.345600 4337 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790243735634835-388-0/minicluster-data/ts-2/wal/instance:
uuid: "f580ddf34a8a4d58ac1f65e7b4ed78e3"
format_stamp: "Formatted at 2026-09-24 09:56:10 on dist-test-slave-w8d2"
server_key: "6e9db5fa5d8a6c0818fc4490c4ec93f0"
server_key_iv: "228288e6d129a683e1d3b33a1eca3595"
server_key_version: "encryptionkey@0"
I20260924 09:56:10.350312 4337 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260924 09:56:10.353396 4352 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 09:56:10.354590 4337 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 09:56:10.354710 4337 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790243735634835-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790243735634835-388-0/minicluster-data/ts-2/wal
uuid: "f580ddf34a8a4d58ac1f65e7b4ed78e3"
format_stamp: "Formatted at 2026-09-24 09:56:10 on dist-test-slave-w8d2"
server_key: "6e9db5fa5d8a6c0818fc4490c4ec93f0"
server_key_iv: "228288e6d129a683e1d3b33a1eca3595"
server_key_version: "encryptionkey@0"
I20260924 09:56:10.354861 4337 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790243735634835-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790243735634835-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790243735634835-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 09:56:10.372648 4337 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 09:56:10.376411 4337 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 09:56:10.376621 4337 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 09:56:10.377400 4337 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 09:56:10.378647 4337 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 09:56:10.378726 4337 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 09:56:10.378810 4337 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 09:56:10.378850 4337 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 09:56:10.406443 4337 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:38659
I20260924 09:56:10.406486 4464 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:38659 every 8 connection(s)
I20260924 09:56:10.407590 4337 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790243735634835-388-0/minicluster-data/ts-2/data/info.pb
I20260924 09:56:10.409071 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu as pid 4337
I20260924 09:56:10.409238 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790243735634835-388-0/minicluster-data/ts-2/wal/instance
I20260924 09:56:10.409703 388 external_mini_cluster.cc:1428] Setting key 44b79fd077a0462232d66ebaeec6b9da
Sep 24 09:56:10 dist-test-slave-w8d2 krb5kdc[3964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243770, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 09:56:10.425689 4200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:10.410068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:58137 (local address 127.0.97.62:44591)
0924 09:56:10.410259 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0924 09:56:10.410265 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 09:56:10.411445 (+ 1180us) server_negotiation.cc:407] Connection header received
0924 09:56:10.413438 (+ 1993us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:56:10.413443 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:56:10.413519 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:56:10.413607 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:56:10.415030 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:10.415788 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:10.417227 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:10.417612 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:10.417715 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:10.420751 (+ 3036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:56:10.420756 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:56:10.420759 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:56:10.420788 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:56:10.422946 (+ 2158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:56:10.423677 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:56:10.423680 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:56:10.423682 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:56:10.423791 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:56:10.424221 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:56:10.424226 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:56:10.424228 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:56:10.424613 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:56:10.424723 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0924 09:56:10.425296 (+ 573us) server_negotiation.cc:299] Negotiation successful
0924 09:56:10.425463 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 09:56:10.426541 4467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:10.410599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44591 (local address 127.0.97.3:58137)
0924 09:56:10.411308 (+ 709us) negotiation.cc:107] Waiting for socket to connect
0924 09:56:10.411355 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0924 09:56:10.413205 (+ 1850us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:56:10.413837 (+ 632us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:56:10.413845 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:56:10.414338 (+ 493us) client_negotiation.cc:189] Negotiated authn=SASL
0924 09:56:10.414859 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:10.414878 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:10.415933 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:10.415939 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:10.417103 (+ 1164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:10.417114 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:10.417812 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:10.417819 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:10.418048 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:10.418589 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 09:56:10.418613 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 09:56:10.420569 (+ 1956us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 09:56:10.423060 (+ 2491us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 09:56:10.423066 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 09:56:10.423076 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0924 09:56:10.423583 (+ 507us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 09:56:10.423885 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 09:56:10.423889 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 09:56:10.423891 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 09:56:10.424108 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 09:56:10.424721 (+ 613us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 09:56:10.424727 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 09:56:10.425008 (+ 281us) client_negotiation.cc:769] Sending connection context
0924 09:56:10.425227 (+ 219us) client_negotiation.cc:240] Negotiation successful
0924 09:56:10.425553 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":271,"threads_started":1}
I20260924 09:56:10.427655 4465 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44591
I20260924 09:56:10.427899 4465 heartbeater.cc:464] Registering TS with master...
I20260924 09:56:10.428514 4465 heartbeater.cc:511] Master 127.0.97.62:44591 requested a full tablet report, sending...
I20260924 09:56:10.429739 4009 ts_manager.cc:194] Registered new tserver with Master: f580ddf34a8a4d58ac1f65e7b4ed78e3 (127.0.97.3:38659)
I20260924 09:56:10.430500 4009 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:58137
I20260924 09:56:10.436322 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 09:56:10 dist-test-slave-w8d2 krb5kdc[3964](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243770, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 09:56:10 dist-test-slave-w8d2 krb5kdc[3964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243770, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260924 09:56:10.476773 4472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:10.463539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35404 (local address 127.0.97.1:33419)
0924 09:56:10.464055 (+ 516us) server_negotiation.cc:206] Beginning negotiation
0924 09:56:10.464063 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 09:56:10.464089 (+ 26us) server_negotiation.cc:407] Connection header received
0924 09:56:10.464210 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:56:10.464216 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:56:10.464440 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:56:10.464578 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:56:10.465845 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:10.466801 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:10.468015 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:10.468522 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:10.468610 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:10.472076 (+ 3466us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:56:10.472087 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:56:10.472100 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:56:10.472129 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:56:10.474655 (+ 2526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:56:10.475168 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:56:10.475174 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:56:10.475177 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:56:10.475311 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:56:10.475593 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:56:10.475596 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:56:10.475598 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:56:10.475880 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:56:10.476104 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0924 09:56:10.476272 (+ 168us) server_negotiation.cc:299] Negotiation successful
0924 09:56:10.476444 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":384,"thread_start_us":247,"threads_started":1}
Sep 24 09:56:10 dist-test-slave-w8d2 krb5kdc[3964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243770, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 09:56:10.492408 4200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:10.480572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43678 (local address 127.0.97.62:44591)
0924 09:56:10.480752 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0924 09:56:10.480758 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 09:56:10.480894 (+ 136us) server_negotiation.cc:407] Connection header received
0924 09:56:10.481067 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:56:10.481071 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:56:10.481128 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:56:10.481205 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:56:10.482337 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:10.483034 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:10.484286 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:10.484674 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:10.484780 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:10.488197 (+ 3417us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:56:10.488203 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:56:10.488206 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:56:10.488238 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:56:10.490334 (+ 2096us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:56:10.490867 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:56:10.490871 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:56:10.490875 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:56:10.490982 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:56:10.491259 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:56:10.491263 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:56:10.491266 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:56:10.491604 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:56:10.491779 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0924 09:56:10.492020 (+ 241us) server_negotiation.cc:299] Negotiation successful
0924 09:56:10.492169 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
W20260924 09:56:10.492893 4009 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43678
I20260924 09:56:10.493794 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu pid 4070
I20260924 09:56:10.505607 4472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:10.494674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35410 (local address 127.0.97.1:33419)
0924 09:56:10.494873 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0924 09:56:10.494879 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 09:56:10.494988 (+ 109us) server_negotiation.cc:407] Connection header received
0924 09:56:10.495146 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:56:10.495151 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:56:10.495212 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:56:10.495317 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:56:10.496391 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:10.497078 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:10.498237 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:10.498667 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:10.498774 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:10.500910 (+ 2136us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:56:10.500915 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:56:10.500918 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:56:10.500949 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:56:10.503295 (+ 2346us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:56:10.503876 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:56:10.503882 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:56:10.503885 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:56:10.504000 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:56:10.504327 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:56:10.504332 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:56:10.504336 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:56:10.504589 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:56:10.504812 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0924 09:56:10.505086 (+ 274us) server_negotiation.cc:299] Negotiation successful
0924 09:56:10.505289 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 09:56:10.506209 4193 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 09:56:10.584486 4198 heartbeater.cc:503] Master 127.0.97.62:44591 was elected leader, sending a full tablet report...
I20260924 09:56:10.584892 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu with pid 4070
W20260924 09:56:10.597893 3995 connection.cc:535] server connection from 127.0.97.1:57155 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260924 09:56:10.598094 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu pid 4204
Sep 24 09:56:10 dist-test-slave-w8d2 krb5kdc[3964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243770, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260924 09:56:10.612116 4477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:10.598876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43884 (local address 127.0.97.2:34707)
0924 09:56:10.599367 (+ 491us) server_negotiation.cc:206] Beginning negotiation
0924 09:56:10.599374 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 09:56:10.599398 (+ 24us) server_negotiation.cc:407] Connection header received
0924 09:56:10.599455 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:56:10.599461 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:56:10.599657 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:56:10.599775 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:56:10.600812 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:10.601912 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:10.603112 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:10.603737 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:10.603829 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:10.607543 (+ 3714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:56:10.607553 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:56:10.607568 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:56:10.607598 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:56:10.609952 (+ 2354us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:56:10.610476 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:56:10.610482 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:56:10.610484 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:56:10.610588 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:56:10.610844 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:56:10.610846 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:56:10.610848 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:56:10.611144 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:56:10.611397 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0924 09:56:10.611605 (+ 208us) server_negotiation.cc:299] Negotiation successful
0924 09:56:10.611779 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":228,"threads_started":1}
I20260924 09:56:10.613282 4327 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 09:56:10.692305 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu with pid 4204
I20260924 09:56:10.704463 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu pid 4337
Sep 24 09:56:10 dist-test-slave-w8d2 krb5kdc[3964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243770, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260924 09:56:10.718598 4479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:10.705454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41628 (local address 127.0.97.3:38659)
0924 09:56:10.706044 (+ 590us) server_negotiation.cc:206] Beginning negotiation
0924 09:56:10.706051 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 09:56:10.706079 (+ 28us) server_negotiation.cc:407] Connection header received
0924 09:56:10.706137 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:56:10.706144 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:56:10.706338 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:56:10.706463 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:56:10.707568 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:10.708524 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:10.709666 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:10.710194 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:10.710282 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:10.713937 (+ 3655us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:56:10.713947 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:56:10.713961 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:56:10.713995 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:56:10.716529 (+ 2534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:56:10.717034 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:56:10.717040 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:56:10.717043 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:56:10.717149 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:56:10.717382 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:56:10.717385 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:56:10.717387 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:56:10.717674 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:56:10.717885 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0924 09:56:10.718083 (+ 198us) server_negotiation.cc:299] Negotiation successful
0924 09:56:10.718267 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":455,"thread_start_us":291,"threads_started":1}
I20260924 09:56:10.719937 4460 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 09:56:10.799631 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu with pid 4337
I20260924 09:56:10.811822 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu pid 3977
I20260924 09:56:10.812870 4039 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 09:56:10.909969 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu with pid 3977
2026-09-24T09:56:10Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (3472 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790243735634835-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 09:56:10 dist-test-slave-w8d2 krb5kdc[4484](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 09:56:10 dist-test-slave-w8d2 krb5kdc[4484](info): set up 2 sockets
Sep 24 09:56:10 dist-test-slave-w8d2 krb5kdc[4484](info): commencing operation
krb5kdc: starting...
W20260924 09:56:11.899466 388 mini_kdc.cc:121] Time spent starting KDC: real 0.954s user 0.000s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790243735634835-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790243735634835-388-0/krb5kdc/test-user.keytab.
Sep 24 09:56:12 dist-test-slave-w8d2 krb5kdc[4484](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243772, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T09:56:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T09:56:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790243735634835-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790243735634835-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260924 09:56:12.074472 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
/tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790243735634835-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790243735634835-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790243735634835-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790243735634835-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35235
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790243735634835-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:35235
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790243735634835-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790243735634835-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790243735634835-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 09:56:12.375501 4497 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 09:56:12.375780 4497 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 09:56:12.375910 4497 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 09:56:12.382881 4497 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 09:56:12.382992 4497 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 09:56:12.383065 4497 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 09:56:12.383145 4497 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 09:56:12.383204 4497 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 09:56:12.392791 4497 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790243735634835-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790243735634835-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:35235
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790243735634835-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35235
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790243735634835-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790243735634835-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 09:56:12.393987 4497 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 09:56:12.395220 4497 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 09:56:12.405858 4502 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:56:12.405923 4505 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:56:12.405943 4503 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 09:56:12.406872 4497 server_base.cc:1034] running on GCE node
I20260924 09:56:12.407660 4497 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 09:56:12.408926 4497 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 09:56:12.410113 4497 hybrid_clock.cc:648] HybridClock initialized: now 1790243772410059 us; error 43 us; skew 500 ppm
I20260924 09:56:12.410435 4497 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 09:56:12 dist-test-slave-w8d2 krb5kdc[4484](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243772, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 09:56:12.413204 4497 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260924 09:56:12.414209 4497 fs_manager.cc:359] Metadata directory not provided
I20260924 09:56:12.414301 4497 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 09:56:12.414579 4497 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 09:56:12.417404 4497 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790243735634835-388-0/minicluster-data/master-0/data/instance:
uuid: "9c917eb636544e09ad3eec94c30189fd"
format_stamp: "Formatted at 2026-09-24 09:56:12 on dist-test-slave-w8d2"
server_key: "6b2a86a585cf530817045ff3afbc3d1e"
server_key_iv: "981992a6a6e083be9626dd6507d842e9"
server_key_version: "encryptionkey@0"
I20260924 09:56:12.418160 4497 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790243735634835-388-0/minicluster-data/master-0/wal/instance:
uuid: "9c917eb636544e09ad3eec94c30189fd"
format_stamp: "Formatted at 2026-09-24 09:56:12 on dist-test-slave-w8d2"
server_key: "6b2a86a585cf530817045ff3afbc3d1e"
server_key_iv: "981992a6a6e083be9626dd6507d842e9"
server_key_version: "encryptionkey@0"
I20260924 09:56:12.422789 4497 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20260924 09:56:12.426051 4511 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 09:56:12.427381 4497 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 09:56:12.427527 4497 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790243735634835-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790243735634835-388-0/minicluster-data/master-0/wal
uuid: "9c917eb636544e09ad3eec94c30189fd"
format_stamp: "Formatted at 2026-09-24 09:56:12 on dist-test-slave-w8d2"
server_key: "6b2a86a585cf530817045ff3afbc3d1e"
server_key_iv: "981992a6a6e083be9626dd6507d842e9"
server_key_version: "encryptionkey@0"
I20260924 09:56:12.427719 4497 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790243735634835-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790243735634835-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790243735634835-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 09:56:12.450408 4497 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 09:56:12.454413 4497 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 09:56:12.454653 4497 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 09:56:12.470021 4497 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:35235
I20260924 09:56:12.469998 4562 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:35235 every 8 connection(s)
I20260924 09:56:12.470847 4497 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790243735634835-388-0/minicluster-data/master-0/data/info.pb
I20260924 09:56:12.474321 4563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 09:56:12.478905 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu as pid 4497
I20260924 09:56:12.479110 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790243735634835-388-0/minicluster-data/master-0/wal/instance
I20260924 09:56:12.479887 388 external_mini_cluster.cc:1428] Setting key 4100ac8fafe579223d2e75d985961734
I20260924 09:56:12.485143 4563 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c917eb636544e09ad3eec94c30189fd: Bootstrap starting.
I20260924 09:56:12.488463 4563 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9c917eb636544e09ad3eec94c30189fd: Neither blocks nor log segments found. Creating new log.
I20260924 09:56:12.489627 4563 log.cc:824] T 00000000000000000000000000000000 P 9c917eb636544e09ad3eec94c30189fd: Log is configured to *not* fsync() on all Append() calls
I20260924 09:56:12.492668 4563 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c917eb636544e09ad3eec94c30189fd: No bootstrap required, opened a new log
Sep 24 09:56:12 dist-test-slave-w8d2 krb5kdc[4484](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243772, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 09:56:12.501520 4566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:12.481790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40194 (local address 127.0.97.62:35235)
0924 09:56:12.482528 (+ 738us) server_negotiation.cc:206] Beginning negotiation
0924 09:56:12.482543 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 09:56:12.482587 (+ 44us) server_negotiation.cc:407] Connection header received
0924 09:56:12.484234 (+ 1647us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:56:12.484255 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:56:12.484640 (+ 385us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:56:12.484944 (+ 304us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:56:12.486589 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:12.487905 (+ 1316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:12.489596 (+ 1691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:12.490495 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:12.490707 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:12.494879 (+ 4172us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:56:12.494886 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:56:12.494900 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:56:12.494939 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:56:12.497785 (+ 2846us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:56:12.498482 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:56:12.498489 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:56:12.498495 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:56:12.498651 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:56:12.499027 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:56:12.499030 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:56:12.499032 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:56:12.499576 (+ 544us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:56:12.499805 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0924 09:56:12.500181 (+ 376us) server_negotiation.cc:299] Negotiation successful
0924 09:56:12.500534 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":527,"thread_start_us":327,"threads_started":1}
I20260924 09:56:12.501324 4563 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9c917eb636544e09ad3eec94c30189fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c917eb636544e09ad3eec94c30189fd" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35235 } }
I20260924 09:56:12.501677 4563 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9c917eb636544e09ad3eec94c30189fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 09:56:12.501775 4563 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9c917eb636544e09ad3eec94c30189fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c917eb636544e09ad3eec94c30189fd, State: Initialized, Role: FOLLOWER
I20260924 09:56:12.502367 4563 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9c917eb636544e09ad3eec94c30189fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c917eb636544e09ad3eec94c30189fd" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35235 } }
I20260924 09:56:12.502553 4563 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9c917eb636544e09ad3eec94c30189fd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 09:56:12.502676 4563 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9c917eb636544e09ad3eec94c30189fd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 09:56:12.502830 4563 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9c917eb636544e09ad3eec94c30189fd [term 0 FOLLOWER]: Advancing to term 1
I20260924 09:56:12.505067 4563 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9c917eb636544e09ad3eec94c30189fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c917eb636544e09ad3eec94c30189fd" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35235 } }
I20260924 09:56:12.505517 4563 leader_election.cc:302] T 00000000000000000000000000000000 P 9c917eb636544e09ad3eec94c30189fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9c917eb636544e09ad3eec94c30189fd; no voters:
I20260924 09:56:12.506012 4563 leader_election.cc:288] T 00000000000000000000000000000000 P 9c917eb636544e09ad3eec94c30189fd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 09:56:12.506131 4568 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9c917eb636544e09ad3eec94c30189fd [term 1 FOLLOWER]: Leader election won for term 1
I20260924 09:56:12.506692 4568 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9c917eb636544e09ad3eec94c30189fd [term 1 LEADER]: Becoming Leader. State: Replica: 9c917eb636544e09ad3eec94c30189fd, State: Running, Role: LEADER
I20260924 09:56:12.507130 4568 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9c917eb636544e09ad3eec94c30189fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c917eb636544e09ad3eec94c30189fd" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35235 } }
I20260924 09:56:12.509086 4568 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c917eb636544e09ad3eec94c30189fd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9c917eb636544e09ad3eec94c30189fd. Latest consensus state: current_term: 1 leader_uuid: "9c917eb636544e09ad3eec94c30189fd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c917eb636544e09ad3eec94c30189fd" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35235 } } }
I20260924 09:56:12.509353 4568 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c917eb636544e09ad3eec94c30189fd [sys.catalog]: This master's current role is: LEADER
I20260924 09:56:12.510038 4570 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c917eb636544e09ad3eec94c30189fd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9c917eb636544e09ad3eec94c30189fd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c917eb636544e09ad3eec94c30189fd" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35235 } } }
I20260924 09:56:12.510370 4570 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c917eb636544e09ad3eec94c30189fd [sys.catalog]: This master's current role is: LEADER
I20260924 09:56:12.512867 4563 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9c917eb636544e09ad3eec94c30189fd [sys.catalog]: configured and running, proceeding with master startup.
I20260924 09:56:12.513134 4572 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 09:56:12.517807 4572 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 09:56:12.525427 4572 catalog_manager.cc:1347] Generated new cluster ID: c09b00c69f21471b85eac55be4d3c523
I20260924 09:56:12.525530 4572 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 09:56:12.550241 4572 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 09:56:12.551573 4572 catalog_manager.cc:1504] Loading token signing keys...
I20260924 09:56:12.558323 4572 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9c917eb636544e09ad3eec94c30189fd: Generated new TSK 0
I20260924 09:56:12.559095 4572 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790243735634835-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790243735634835-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260924 09:56:12.598948 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
/tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790243735634835-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790243735634835-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790243735634835-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790243735634835-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35235
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790243735634835-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790243735634835-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790243735634835-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790243735634835-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 09:56:12.903724 4589 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 09:56:12.903976 4589 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 09:56:12.904104 4589 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 09:56:12.911094 4589 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 09:56:12.911190 4589 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 09:56:12.911355 4589 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260924 09:56:12.920549 4589 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790243735634835-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790243735634835-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790243735634835-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790243735634835-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35235
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790243735634835-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 09:56:12.921751 4589 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 09:56:12.922986 4589 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 09:56:12.935272 4597 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:56:12.935290 4594 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:56:12.935295 4595 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 09:56:12.935933 4589 server_base.cc:1034] running on GCE node
I20260924 09:56:12.936460 4589 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 09:56:12.937292 4589 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 09:56:12.938483 4589 hybrid_clock.cc:648] HybridClock initialized: now 1790243772938434 us; error 43 us; skew 500 ppm
I20260924 09:56:12.938807 4589 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 09:56:12 dist-test-slave-w8d2 krb5kdc[4484](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243772, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 09:56:12.941715 4589 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260924 09:56:12.942728 4589 fs_manager.cc:359] Metadata directory not provided
I20260924 09:56:12.942812 4589 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 09:56:12.943105 4589 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 09:56:12.945988 4589 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790243735634835-388-0/minicluster-data/ts-0/data/instance:
uuid: "467506185daf40dc942de690005ce088"
format_stamp: "Formatted at 2026-09-24 09:56:12 on dist-test-slave-w8d2"
server_key: "c9983d87fc167372eb89373e967d3aef"
server_key_iv: "cb46d862d6b014925d2bbc2c7de049bb"
server_key_version: "encryptionkey@0"
I20260924 09:56:12.946755 4589 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790243735634835-388-0/minicluster-data/ts-0/wal/instance:
uuid: "467506185daf40dc942de690005ce088"
format_stamp: "Formatted at 2026-09-24 09:56:12 on dist-test-slave-w8d2"
server_key: "c9983d87fc167372eb89373e967d3aef"
server_key_iv: "cb46d862d6b014925d2bbc2c7de049bb"
server_key_version: "encryptionkey@0"
I20260924 09:56:12.951470 4589 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.003s
I20260924 09:56:12.954670 4603 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 09:56:12.955952 4589 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 09:56:12.956094 4589 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790243735634835-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790243735634835-388-0/minicluster-data/ts-0/wal
uuid: "467506185daf40dc942de690005ce088"
format_stamp: "Formatted at 2026-09-24 09:56:12 on dist-test-slave-w8d2"
server_key: "c9983d87fc167372eb89373e967d3aef"
server_key_iv: "cb46d862d6b014925d2bbc2c7de049bb"
server_key_version: "encryptionkey@0"
I20260924 09:56:12.956278 4589 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790243735634835-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790243735634835-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790243735634835-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 09:56:12.980170 4589 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 09:56:12.984349 4589 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 09:56:12.984643 4589 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 09:56:12.985548 4589 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 09:56:12.986769 4589 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 09:56:12.986863 4589 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 09:56:12.986965 4589 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 09:56:12.987031 4589 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 09:56:13.014796 4589 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:35417
I20260924 09:56:13.014843 4715 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:35417 every 8 connection(s)
I20260924 09:56:13.015650 4589 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790243735634835-388-0/minicluster-data/ts-0/data/info.pb
I20260924 09:56:13.025584 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu as pid 4589
I20260924 09:56:13.025758 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790243735634835-388-0/minicluster-data/ts-0/wal/instance
I20260924 09:56:13.026333 388 external_mini_cluster.cc:1428] Setting key e3b217add63c5958c1a31d14bc5710c5
Sep 24 09:56:13 dist-test-slave-w8d2 krb5kdc[4484](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243772, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 09:56:13.034353 4718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:13.017925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:51849 (local address 127.0.97.62:35235)
0924 09:56:13.018321 (+ 396us) server_negotiation.cc:206] Beginning negotiation
0924 09:56:13.018326 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 09:56:13.019260 (+ 934us) server_negotiation.cc:407] Connection header received
0924 09:56:13.020981 (+ 1721us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:56:13.020987 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:56:13.021067 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:56:13.021162 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:56:13.022587 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:13.023344 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:13.024818 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:13.025279 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:13.025407 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:13.028958 (+ 3551us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:56:13.028963 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:56:13.028966 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:56:13.028995 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:56:13.031421 (+ 2426us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:56:13.032265 (+ 844us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:56:13.032270 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:56:13.032272 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:56:13.032392 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:56:13.032795 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:56:13.032798 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:56:13.032800 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:56:13.033045 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:56:13.033181 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0924 09:56:13.033787 (+ 606us) server_negotiation.cc:299] Negotiation successful
0924 09:56:13.034015 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295,"thread_start_us":220,"threads_started":1}
I20260924 09:56:13.035075 4719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:13.018376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35235 (local address 127.0.97.1:51849)
0924 09:56:13.019074 (+ 698us) negotiation.cc:107] Waiting for socket to connect
0924 09:56:13.019116 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 09:56:13.020728 (+ 1612us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:56:13.021395 (+ 667us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:56:13.021403 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:56:13.021882 (+ 479us) client_negotiation.cc:189] Negotiated authn=SASL
0924 09:56:13.022428 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:13.022446 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:13.023475 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:13.023479 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:13.024688 (+ 1209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:13.024700 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:13.025487 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:13.025493 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:13.025708 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:13.026317 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 09:56:13.026338 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 09:56:13.028754 (+ 2416us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 09:56:13.031551 (+ 2797us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 09:56:13.031559 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 09:56:13.031575 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 09:56:13.032111 (+ 536us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 09:56:13.032500 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 09:56:13.032504 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 09:56:13.032508 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 09:56:13.032707 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 09:56:13.033157 (+ 450us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 09:56:13.033171 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 09:56:13.033466 (+ 295us) client_negotiation.cc:769] Sending connection context
0924 09:56:13.033674 (+ 208us) client_negotiation.cc:240] Negotiation successful
0924 09:56:13.034033 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":295,"threads_started":1}
I20260924 09:56:13.036494 4716 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35235
I20260924 09:56:13.036762 4716 heartbeater.cc:464] Registering TS with master...
I20260924 09:56:13.037343 4716 heartbeater.cc:511] Master 127.0.97.62:35235 requested a full tablet report, sending...
I20260924 09:56:13.039208 4528 ts_manager.cc:194] Registered new tserver with Master: 467506185daf40dc942de690005ce088 (127.0.97.1:35417)
I20260924 09:56:13.040647 4528 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:51849
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790243735634835-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790243735634835-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260924 09:56:13.064675 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
/tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790243735634835-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790243735634835-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790243735634835-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790243735634835-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35235
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790243735634835-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790243735634835-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790243735634835-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790243735634835-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 09:56:13.382450 4722 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 09:56:13.382736 4722 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 09:56:13.382887 4722 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 09:56:13.390139 4722 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 09:56:13.390251 4722 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 09:56:13.390398 4722 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260924 09:56:13.399957 4722 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790243735634835-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790243735634835-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790243735634835-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790243735634835-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35235
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790243735634835-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 09:56:13.401188 4722 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 09:56:13.402434 4722 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 09:56:13.414410 4727 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:56:13.414382 4728 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:56:13.414440 4730 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 09:56:13.415057 4722 server_base.cc:1034] running on GCE node
I20260924 09:56:13.415670 4722 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 09:56:13.416569 4722 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 09:56:13.417759 4722 hybrid_clock.cc:648] HybridClock initialized: now 1790243773417700 us; error 57 us; skew 500 ppm
I20260924 09:56:13.418045 4722 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 09:56:13 dist-test-slave-w8d2 krb5kdc[4484](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243773, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 09:56:13.420915 4722 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260924 09:56:13.421914 4722 fs_manager.cc:359] Metadata directory not provided
I20260924 09:56:13.421996 4722 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 09:56:13.422251 4722 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 09:56:13.425000 4722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790243735634835-388-0/minicluster-data/ts-1/data/instance:
uuid: "29f47496f0844e8cb35eec7c187794e5"
format_stamp: "Formatted at 2026-09-24 09:56:13 on dist-test-slave-w8d2"
server_key: "a7d2b28a87520e70665cd12b20739158"
server_key_iv: "d1602535f3bc4cdd89a3d95d5ad399cd"
server_key_version: "encryptionkey@0"
I20260924 09:56:13.425704 4722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790243735634835-388-0/minicluster-data/ts-1/wal/instance:
uuid: "29f47496f0844e8cb35eec7c187794e5"
format_stamp: "Formatted at 2026-09-24 09:56:13 on dist-test-slave-w8d2"
server_key: "a7d2b28a87520e70665cd12b20739158"
server_key_iv: "d1602535f3bc4cdd89a3d95d5ad399cd"
server_key_version: "encryptionkey@0"
I20260924 09:56:13.430435 4722 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260924 09:56:13.433655 4736 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 09:56:13.434834 4722 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260924 09:56:13.434955 4722 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790243735634835-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790243735634835-388-0/minicluster-data/ts-1/wal
uuid: "29f47496f0844e8cb35eec7c187794e5"
format_stamp: "Formatted at 2026-09-24 09:56:13 on dist-test-slave-w8d2"
server_key: "a7d2b28a87520e70665cd12b20739158"
server_key_iv: "d1602535f3bc4cdd89a3d95d5ad399cd"
server_key_version: "encryptionkey@0"
I20260924 09:56:13.435114 4722 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790243735634835-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790243735634835-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790243735634835-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 09:56:13.450477 4722 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 09:56:13.454032 4722 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 09:56:13.454267 4722 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 09:56:13.455113 4722 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 09:56:13.456344 4722 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 09:56:13.456429 4722 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 09:56:13.456526 4722 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 09:56:13.456595 4722 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 09:56:13.485891 4722 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:37345
I20260924 09:56:13.485922 4848 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:37345 every 8 connection(s)
I20260924 09:56:13.486668 4722 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790243735634835-388-0/minicluster-data/ts-1/data/info.pb
I20260924 09:56:13.490208 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu as pid 4722
I20260924 09:56:13.490386 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790243735634835-388-0/minicluster-data/ts-1/wal/instance
I20260924 09:56:13.490916 388 external_mini_cluster.cc:1428] Setting key 8df898a0ad78245a4c76fb010a59bb72
Sep 24 09:56:13 dist-test-slave-w8d2 krb5kdc[4484](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243773, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 09:56:13.505798 4718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:13.488831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:57751 (local address 127.0.97.62:35235)
0924 09:56:13.488983 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0924 09:56:13.488987 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 09:56:13.490181 (+ 1194us) server_negotiation.cc:407] Connection header received
0924 09:56:13.492279 (+ 2098us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:56:13.492283 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:56:13.492374 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:56:13.492477 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:56:13.494301 (+ 1824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:13.495000 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:13.496993 (+ 1993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:13.497379 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:13.497467 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:13.500857 (+ 3390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:56:13.500861 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:56:13.500864 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:56:13.500894 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:56:13.503063 (+ 2169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:56:13.503738 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:56:13.503742 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:56:13.503744 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:56:13.503854 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:56:13.504269 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:56:13.504272 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:56:13.504274 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:56:13.504509 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:56:13.504652 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0924 09:56:13.505280 (+ 628us) server_negotiation.cc:299] Negotiation successful
0924 09:56:13.505495 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 09:56:13.506804 4851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:13.489310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35235 (local address 127.0.97.2:57751)
0924 09:56:13.489995 (+ 685us) negotiation.cc:107] Waiting for socket to connect
0924 09:56:13.490051 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0924 09:56:13.492034 (+ 1983us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:56:13.492701 (+ 667us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:56:13.492712 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:56:13.493398 (+ 686us) client_negotiation.cc:189] Negotiated authn=SASL
0924 09:56:13.494122 (+ 724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:13.494146 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:13.495133 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:13.495139 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:13.496883 (+ 1744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:13.496896 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:13.497569 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:13.497575 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:13.497809 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:13.498480 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 09:56:13.498503 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 09:56:13.500661 (+ 2158us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 09:56:13.503184 (+ 2523us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 09:56:13.503193 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 09:56:13.503209 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 09:56:13.503623 (+ 414us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 09:56:13.503948 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 09:56:13.503953 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 09:56:13.503957 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 09:56:13.504163 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 09:56:13.504634 (+ 471us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 09:56:13.504640 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 09:56:13.504962 (+ 322us) client_negotiation.cc:769] Sending connection context
0924 09:56:13.505180 (+ 218us) client_negotiation.cc:240] Negotiation successful
0924 09:56:13.505668 (+ 488us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":260,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260924 09:56:13.508167 4849 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35235
I20260924 09:56:13.508442 4849 heartbeater.cc:464] Registering TS with master...
I20260924 09:56:13.509050 4849 heartbeater.cc:511] Master 127.0.97.62:35235 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260924 09:56:13.510351 4528 ts_manager.cc:194] Registered new tserver with Master: 29f47496f0844e8cb35eec7c187794e5 (127.0.97.2:37345)
I20260924 09:56:13.511078 4528 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:57751
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790243735634835-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790243735634835-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260924 09:56:13.545910 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
/tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790243735634835-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790243735634835-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790243735634835-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790243735634835-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35235
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790243735634835-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790243735634835-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790243735634835-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790243735634835-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 09:56:13.849999 4854 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 09:56:13.850266 4854 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 09:56:13.850359 4854 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 09:56:13.857232 4854 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 09:56:13.857337 4854 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 09:56:13.857453 4854 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260924 09:56:13.866693 4854 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790243735634835-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790243735634835-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790243735634835-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790243735634835-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35235
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790243735634835-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 09:56:13.867878 4854 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 09:56:13.869082 4854 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 09:56:13.881017 4859 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:56:13.881017 4860 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:56:13.881206 4862 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 09:56:13.881685 4854 server_base.cc:1034] running on GCE node
I20260924 09:56:13.882234 4854 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 09:56:13.883077 4854 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 09:56:13.884277 4854 hybrid_clock.cc:648] HybridClock initialized: now 1790243773884209 us; error 59 us; skew 500 ppm
I20260924 09:56:13.884552 4854 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 09:56:13 dist-test-slave-w8d2 krb5kdc[4484](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243773, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 09:56:13.887529 4854 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260924 09:56:13.888618 4854 fs_manager.cc:359] Metadata directory not provided
I20260924 09:56:13.888705 4854 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 09:56:13.888967 4854 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 09:56:13.891827 4854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790243735634835-388-0/minicluster-data/ts-2/data/instance:
uuid: "50fbe96f069f4ac2af2133a074ab8e5f"
format_stamp: "Formatted at 2026-09-24 09:56:13 on dist-test-slave-w8d2"
server_key: "0c54b73f894fde3597df53c8d1932571"
server_key_iv: "d9cfdfd55ea35546cd4ebae2e4ebfaa5"
server_key_version: "encryptionkey@0"
I20260924 09:56:13.892581 4854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790243735634835-388-0/minicluster-data/ts-2/wal/instance:
uuid: "50fbe96f069f4ac2af2133a074ab8e5f"
format_stamp: "Formatted at 2026-09-24 09:56:13 on dist-test-slave-w8d2"
server_key: "0c54b73f894fde3597df53c8d1932571"
server_key_iv: "d9cfdfd55ea35546cd4ebae2e4ebfaa5"
server_key_version: "encryptionkey@0"
I20260924 09:56:13.897369 4854 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 09:56:13.900601 4868 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 09:56:13.901857 4854 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 09:56:13.901983 4854 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790243735634835-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790243735634835-388-0/minicluster-data/ts-2/wal
uuid: "50fbe96f069f4ac2af2133a074ab8e5f"
format_stamp: "Formatted at 2026-09-24 09:56:13 on dist-test-slave-w8d2"
server_key: "0c54b73f894fde3597df53c8d1932571"
server_key_iv: "d9cfdfd55ea35546cd4ebae2e4ebfaa5"
server_key_version: "encryptionkey@0"
I20260924 09:56:13.902149 4854 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790243735634835-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790243735634835-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790243735634835-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 09:56:13.918622 4854 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 09:56:13.922526 4854 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 09:56:13.922735 4854 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 09:56:13.923568 4854 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 09:56:13.924733 4854 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 09:56:13.924815 4854 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 09:56:13.924893 4854 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 09:56:13.924944 4854 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 09:56:13.956286 4854 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:35703
I20260924 09:56:13.956321 4980 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:35703 every 8 connection(s)
I20260924 09:56:13.957057 4854 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790243735634835-388-0/minicluster-data/ts-2/data/info.pb
I20260924 09:56:13.960996 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu as pid 4854
I20260924 09:56:13.961159 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790243735634835-388-0/minicluster-data/ts-2/wal/instance
I20260924 09:56:13.961658 388 external_mini_cluster.cc:1428] Setting key 267e9d15a365f41fbdf579e2fbb90f5b
Sep 24 09:56:13 dist-test-slave-w8d2 krb5kdc[4484](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243773, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 09:56:13.974890 4718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:13.959381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:57621 (local address 127.0.97.62:35235)
0924 09:56:13.959530 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0924 09:56:13.959534 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 09:56:13.960698 (+ 1164us) server_negotiation.cc:407] Connection header received
0924 09:56:13.962954 (+ 2256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:56:13.962959 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:56:13.963038 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:56:13.963119 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:56:13.964812 (+ 1693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:13.965531 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:13.966923 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:13.967360 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:13.967448 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:13.970429 (+ 2981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:56:13.970433 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:56:13.970436 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:56:13.970466 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:56:13.972586 (+ 2120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:56:13.973183 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:56:13.973186 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:56:13.973189 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:56:13.973284 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:56:13.973598 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:56:13.973601 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:56:13.973602 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:56:13.973834 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:56:13.973941 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0924 09:56:13.974496 (+ 555us) server_negotiation.cc:299] Negotiation successful
0924 09:56:13.974635 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 09:56:13.976047 4983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:13.959891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35235 (local address 127.0.97.3:57621)
0924 09:56:13.960537 (+ 646us) negotiation.cc:107] Waiting for socket to connect
0924 09:56:13.960576 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 09:56:13.962681 (+ 2105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:56:13.963405 (+ 724us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:56:13.963416 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:56:13.963983 (+ 567us) client_negotiation.cc:189] Negotiated authn=SASL
0924 09:56:13.964645 (+ 662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:13.964686 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:13.965637 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:13.965640 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:13.966818 (+ 1178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:13.966828 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:13.967517 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:13.967521 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:13.967699 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:13.968255 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 09:56:13.968275 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 09:56:13.970219 (+ 1944us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 09:56:13.972695 (+ 2476us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 09:56:13.972701 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 09:56:13.972712 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0924 09:56:13.973103 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 09:56:13.973355 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 09:56:13.973358 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 09:56:13.973360 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 09:56:13.973528 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 09:56:13.973921 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 09:56:13.973926 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 09:56:13.974201 (+ 275us) client_negotiation.cc:769] Sending connection context
0924 09:56:13.974434 (+ 233us) client_negotiation.cc:240] Negotiation successful
0924 09:56:13.974818 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":254,"threads_started":1}
I20260924 09:56:13.977228 4981 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35235
I20260924 09:56:13.977492 4981 heartbeater.cc:464] Registering TS with master...
I20260924 09:56:13.978050 4981 heartbeater.cc:511] Master 127.0.97.62:35235 requested a full tablet report, sending...
I20260924 09:56:13.979266 4528 ts_manager.cc:194] Registered new tserver with Master: 50fbe96f069f4ac2af2133a074ab8e5f (127.0.97.3:35703)
I20260924 09:56:13.980011 4528 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:57621
I20260924 09:56:13.988361 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 09:56:14.002739 4718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:13.992405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58660 (local address 127.0.97.62:35235)
0924 09:56:13.992548 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0924 09:56:13.992552 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 09:56:13.992969 (+ 417us) server_negotiation.cc:407] Connection header received
0924 09:56:13.993089 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:56:13.993092 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:56:13.993143 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:56:13.993214 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:56:13.994429 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:13.995085 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:13.996201 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:13.996595 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:13.996684 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:13.998575 (+ 1891us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:56:13.998580 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:56:13.998583 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:56:13.998610 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:56:14.000880 (+ 2270us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:56:14.001389 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:56:14.001392 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:56:14.001395 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:56:14.001493 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:56:14.001733 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:56:14.001736 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:56:14.001738 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:56:14.002006 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:56:14.002170 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0924 09:56:14.002379 (+ 209us) server_negotiation.cc:299] Negotiation successful
0924 09:56:14.002532 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 09:56:14.006307 4528 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58660:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 09:56:14.012954 4528 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 09:56:14.036128 4995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:14.025123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48354 (local address 127.0.97.2:37345)
0924 09:56:14.026844 (+ 1721us) server_negotiation.cc:206] Beginning negotiation
0924 09:56:14.026853 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 09:56:14.027518 (+ 665us) server_negotiation.cc:407] Connection header received
0924 09:56:14.027902 (+ 384us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:56:14.027908 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:56:14.028153 (+ 245us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:56:14.028312 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 09:56:14.029540 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:14.030986 (+ 1446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:14.033590 (+ 2604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:14.034977 (+ 1387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:14.035101 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:14.035326 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0924 09:56:14.035648 (+ 322us) server_negotiation.cc:299] Negotiation successful
0924 09:56:14.035820 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1598,"thread_start_us":1485,"threads_started":1}
I20260924 09:56:14.036314 4996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:14.025630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59956 (local address 127.0.97.3:35703)
0924 09:56:14.026469 (+ 839us) server_negotiation.cc:206] Beginning negotiation
0924 09:56:14.026480 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 09:56:14.027517 (+ 1037us) server_negotiation.cc:407] Connection header received
0924 09:56:14.027893 (+ 376us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:56:14.027901 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:56:14.028155 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:56:14.028312 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 09:56:14.030101 (+ 1789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:14.031552 (+ 1451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:14.033793 (+ 2241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:14.035091 (+ 1298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:14.035194 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:14.035410 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0924 09:56:14.035849 (+ 439us) server_negotiation.cc:299] Negotiation successful
0924 09:56:14.036005 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":627,"thread_start_us":451,"threads_started":1}
I20260924 09:56:14.037093 4993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:14.025038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:37345 (local address 127.0.0.1:48354)
0924 09:56:14.027355 (+ 2317us) negotiation.cc:107] Waiting for socket to connect
0924 09:56:14.027422 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0924 09:56:14.027664 (+ 242us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:56:14.028594 (+ 930us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:56:14.028601 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:56:14.028639 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 09:56:14.029084 (+ 445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:14.029097 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:14.031090 (+ 1993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:14.031095 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:14.033454 (+ 2359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:14.033466 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:14.035165 (+ 1699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:14.035171 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:14.035395 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:14.035447 (+ 52us) client_negotiation.cc:769] Sending connection context
0924 09:56:14.035586 (+ 139us) client_negotiation.cc:240] Negotiation successful
0924 09:56:14.035787 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2055,"spinlock_wait_cycles":11008,"thread_start_us":1954,"threads_started":1}
I20260924 09:56:14.037110 4994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:14.025086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:35703 (local address 127.0.0.1:59956)
0924 09:56:14.027355 (+ 2269us) negotiation.cc:107] Waiting for socket to connect
0924 09:56:14.027422 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0924 09:56:14.027664 (+ 242us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:56:14.028594 (+ 930us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:56:14.028601 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:56:14.028639 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 09:56:14.029863 (+ 1224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:14.029876 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:14.031660 (+ 1784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:14.031665 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:14.033647 (+ 1982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:14.033660 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:14.035519 (+ 1859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:14.035523 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:14.035668 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:14.035697 (+ 29us) client_negotiation.cc:769] Sending connection context
0924 09:56:14.036570 (+ 873us) client_negotiation.cc:240] Negotiation successful
0924 09:56:14.036712 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2039,"mutex_wait_us":560,"spinlock_wait_cycles":19072,"thread_start_us":1859,"threads_started":1}
I20260924 09:56:14.037613 4997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:14.027960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35417 (local address 127.0.0.1:47318)
0924 09:56:14.030169 (+ 2209us) negotiation.cc:107] Waiting for socket to connect
0924 09:56:14.030188 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 09:56:14.030308 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:56:14.030879 (+ 571us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:56:14.030884 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:56:14.030909 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 09:56:14.031360 (+ 451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:14.031372 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:14.033280 (+ 1908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:14.033285 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:14.034707 (+ 1422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:14.034718 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:14.036684 (+ 1966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:14.036690 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:14.036850 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:14.036880 (+ 30us) client_negotiation.cc:769] Sending connection context
0924 09:56:14.036965 (+ 85us) client_negotiation.cc:240] Negotiation successful
0924 09:56:14.037102 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1957,"thread_start_us":1427,"threads_started":1}
I20260924 09:56:14.037949 4998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:14.028531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47318 (local address 127.0.97.1:35417)
0924 09:56:14.030072 (+ 1541us) server_negotiation.cc:206] Beginning negotiation
0924 09:56:14.030081 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 09:56:14.030283 (+ 202us) server_negotiation.cc:407] Connection header received
0924 09:56:14.030451 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:56:14.030459 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:56:14.030694 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:56:14.030861 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 09:56:14.031528 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:14.033140 (+ 1612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:14.034862 (+ 1722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:14.036197 (+ 1335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:14.036331 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:14.036551 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0924 09:56:14.037199 (+ 648us) server_negotiation.cc:299] Negotiation successful
0924 09:56:14.037666 (+ 467us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1378,"thread_start_us":1256,"threads_started":1}
I20260924 09:56:14.041141 4784 tablet_service.cc:1467] Processing CreateTablet for tablet 1f5f65ad44a64307b7a3b8c37cfd749f (DEFAULT_TABLE table=test-table [id=6090a00cf22345f7b96ee30ee1b6ce34]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 09:56:14.041141 4651 tablet_service.cc:1467] Processing CreateTablet for tablet 1f5f65ad44a64307b7a3b8c37cfd749f (DEFAULT_TABLE table=test-table [id=6090a00cf22345f7b96ee30ee1b6ce34]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 09:56:14.042594 4651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f5f65ad44a64307b7a3b8c37cfd749f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 09:56:14.043304 4716 heartbeater.cc:503] Master 127.0.97.62:35235 was elected leader, sending a full tablet report...
I20260924 09:56:14.044365 4916 tablet_service.cc:1467] Processing CreateTablet for tablet 1f5f65ad44a64307b7a3b8c37cfd749f (DEFAULT_TABLE table=test-table [id=6090a00cf22345f7b96ee30ee1b6ce34]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 09:56:14.045717 4916 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f5f65ad44a64307b7a3b8c37cfd749f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 09:56:14.047894 4784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f5f65ad44a64307b7a3b8c37cfd749f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 09:56:14.056584 4999 tablet_bootstrap.cc:492] T 1f5f65ad44a64307b7a3b8c37cfd749f P 467506185daf40dc942de690005ce088: Bootstrap starting.
I20260924 09:56:14.058138 5000 tablet_bootstrap.cc:492] T 1f5f65ad44a64307b7a3b8c37cfd749f P 29f47496f0844e8cb35eec7c187794e5: Bootstrap starting.
I20260924 09:56:14.059598 5001 tablet_bootstrap.cc:492] T 1f5f65ad44a64307b7a3b8c37cfd749f P 50fbe96f069f4ac2af2133a074ab8e5f: Bootstrap starting.
I20260924 09:56:14.060227 4999 tablet_bootstrap.cc:654] T 1f5f65ad44a64307b7a3b8c37cfd749f P 467506185daf40dc942de690005ce088: Neither blocks nor log segments found. Creating new log.
I20260924 09:56:14.061421 5000 tablet_bootstrap.cc:654] T 1f5f65ad44a64307b7a3b8c37cfd749f P 29f47496f0844e8cb35eec7c187794e5: Neither blocks nor log segments found. Creating new log.
I20260924 09:56:14.061486 4999 log.cc:824] T 1f5f65ad44a64307b7a3b8c37cfd749f P 467506185daf40dc942de690005ce088: Log is configured to *not* fsync() on all Append() calls
I20260924 09:56:14.062448 5000 log.cc:824] T 1f5f65ad44a64307b7a3b8c37cfd749f P 29f47496f0844e8cb35eec7c187794e5: Log is configured to *not* fsync() on all Append() calls
I20260924 09:56:14.062697 5001 tablet_bootstrap.cc:654] T 1f5f65ad44a64307b7a3b8c37cfd749f P 50fbe96f069f4ac2af2133a074ab8e5f: Neither blocks nor log segments found. Creating new log.
I20260924 09:56:14.063870 5001 log.cc:824] T 1f5f65ad44a64307b7a3b8c37cfd749f P 50fbe96f069f4ac2af2133a074ab8e5f: Log is configured to *not* fsync() on all Append() calls
I20260924 09:56:14.064735 4999 tablet_bootstrap.cc:492] T 1f5f65ad44a64307b7a3b8c37cfd749f P 467506185daf40dc942de690005ce088: No bootstrap required, opened a new log
I20260924 09:56:14.065014 4999 ts_tablet_manager.cc:1397] T 1f5f65ad44a64307b7a3b8c37cfd749f P 467506185daf40dc942de690005ce088: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.004s
I20260924 09:56:14.066634 5001 tablet_bootstrap.cc:492] T 1f5f65ad44a64307b7a3b8c37cfd749f P 50fbe96f069f4ac2af2133a074ab8e5f: No bootstrap required, opened a new log
I20260924 09:56:14.066633 5000 tablet_bootstrap.cc:492] T 1f5f65ad44a64307b7a3b8c37cfd749f P 29f47496f0844e8cb35eec7c187794e5: No bootstrap required, opened a new log
I20260924 09:56:14.066879 5000 ts_tablet_manager.cc:1397] T 1f5f65ad44a64307b7a3b8c37cfd749f P 29f47496f0844e8cb35eec7c187794e5: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.002s
I20260924 09:56:14.066879 5001 ts_tablet_manager.cc:1397] T 1f5f65ad44a64307b7a3b8c37cfd749f P 50fbe96f069f4ac2af2133a074ab8e5f: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260924 09:56:14.071810 4999 raft_consensus.cc:348] T 1f5f65ad44a64307b7a3b8c37cfd749f P 467506185daf40dc942de690005ce088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f47496f0844e8cb35eec7c187794e5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37345 } } peers { permanent_uuid: "50fbe96f069f4ac2af2133a074ab8e5f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35703 } } peers { permanent_uuid: "467506185daf40dc942de690005ce088" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35417 } }
I20260924 09:56:14.072144 4999 raft_consensus.cc:374] T 1f5f65ad44a64307b7a3b8c37cfd749f P 467506185daf40dc942de690005ce088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 09:56:14.072269 4999 raft_consensus.cc:729] T 1f5f65ad44a64307b7a3b8c37cfd749f P 467506185daf40dc942de690005ce088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 467506185daf40dc942de690005ce088, State: Initialized, Role: FOLLOWER
I20260924 09:56:14.072824 4999 consensus_queue.cc:260] T 1f5f65ad44a64307b7a3b8c37cfd749f P 467506185daf40dc942de690005ce088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f47496f0844e8cb35eec7c187794e5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37345 } } peers { permanent_uuid: "50fbe96f069f4ac2af2133a074ab8e5f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35703 } } peers { permanent_uuid: "467506185daf40dc942de690005ce088" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35417 } }
I20260924 09:56:14.076799 4999 ts_tablet_manager.cc:1428] T 1f5f65ad44a64307b7a3b8c37cfd749f P 467506185daf40dc942de690005ce088: Time spent starting tablet: real 0.012s user 0.009s sys 0.000s
I20260924 09:56:14.076838 5000 raft_consensus.cc:348] T 1f5f65ad44a64307b7a3b8c37cfd749f P 29f47496f0844e8cb35eec7c187794e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f47496f0844e8cb35eec7c187794e5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37345 } } peers { permanent_uuid: "50fbe96f069f4ac2af2133a074ab8e5f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35703 } } peers { permanent_uuid: "467506185daf40dc942de690005ce088" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35417 } }
I20260924 09:56:14.076836 5001 raft_consensus.cc:348] T 1f5f65ad44a64307b7a3b8c37cfd749f P 50fbe96f069f4ac2af2133a074ab8e5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f47496f0844e8cb35eec7c187794e5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37345 } } peers { permanent_uuid: "50fbe96f069f4ac2af2133a074ab8e5f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35703 } } peers { permanent_uuid: "467506185daf40dc942de690005ce088" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35417 } }
I20260924 09:56:14.077237 5000 raft_consensus.cc:374] T 1f5f65ad44a64307b7a3b8c37cfd749f P 29f47496f0844e8cb35eec7c187794e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 09:56:14.077237 5001 raft_consensus.cc:374] T 1f5f65ad44a64307b7a3b8c37cfd749f P 50fbe96f069f4ac2af2133a074ab8e5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 09:56:14.077332 5001 raft_consensus.cc:729] T 1f5f65ad44a64307b7a3b8c37cfd749f P 50fbe96f069f4ac2af2133a074ab8e5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50fbe96f069f4ac2af2133a074ab8e5f, State: Initialized, Role: FOLLOWER
I20260924 09:56:14.077332 5000 raft_consensus.cc:729] T 1f5f65ad44a64307b7a3b8c37cfd749f P 29f47496f0844e8cb35eec7c187794e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29f47496f0844e8cb35eec7c187794e5, State: Initialized, Role: FOLLOWER
I20260924 09:56:14.077901 5000 consensus_queue.cc:260] T 1f5f65ad44a64307b7a3b8c37cfd749f P 29f47496f0844e8cb35eec7c187794e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f47496f0844e8cb35eec7c187794e5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37345 } } peers { permanent_uuid: "50fbe96f069f4ac2af2133a074ab8e5f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35703 } } peers { permanent_uuid: "467506185daf40dc942de690005ce088" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35417 } }
I20260924 09:56:14.077903 5001 consensus_queue.cc:260] T 1f5f65ad44a64307b7a3b8c37cfd749f P 50fbe96f069f4ac2af2133a074ab8e5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f47496f0844e8cb35eec7c187794e5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37345 } } peers { permanent_uuid: "50fbe96f069f4ac2af2133a074ab8e5f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35703 } } peers { permanent_uuid: "467506185daf40dc942de690005ce088" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35417 } }
I20260924 09:56:14.079452 4981 heartbeater.cc:503] Master 127.0.97.62:35235 was elected leader, sending a full tablet report...
I20260924 09:56:14.079597 5001 ts_tablet_manager.cc:1428] T 1f5f65ad44a64307b7a3b8c37cfd749f P 50fbe96f069f4ac2af2133a074ab8e5f: Time spent starting tablet: real 0.013s user 0.014s sys 0.000s
I20260924 09:56:14.079871 5000 ts_tablet_manager.cc:1428] T 1f5f65ad44a64307b7a3b8c37cfd749f P 29f47496f0844e8cb35eec7c187794e5: Time spent starting tablet: real 0.013s user 0.008s sys 0.006s
I20260924 09:56:14.080389 4849 heartbeater.cc:503] Master 127.0.97.62:35235 was elected leader, sending a full tablet report...
I20260924 09:56:14.143121 5005 raft_consensus.cc:482] T 1f5f65ad44a64307b7a3b8c37cfd749f P 467506185daf40dc942de690005ce088 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 09:56:14.143421 5005 raft_consensus.cc:504] T 1f5f65ad44a64307b7a3b8c37cfd749f P 467506185daf40dc942de690005ce088 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f47496f0844e8cb35eec7c187794e5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37345 } } peers { permanent_uuid: "50fbe96f069f4ac2af2133a074ab8e5f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35703 } } peers { permanent_uuid: "467506185daf40dc942de690005ce088" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35417 } }
I20260924 09:56:14.145052 5005 leader_election.cc:288] T 1f5f65ad44a64307b7a3b8c37cfd749f P 467506185daf40dc942de690005ce088 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 29f47496f0844e8cb35eec7c187794e5 (127.0.97.2:37345), 50fbe96f069f4ac2af2133a074ab8e5f (127.0.97.3:35703)
I20260924 09:56:14.151325 4995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:14.145723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:34569 (local address 127.0.97.2:37345)
0924 09:56:14.145922 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0924 09:56:14.145928 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 09:56:14.146296 (+ 368us) server_negotiation.cc:407] Connection header received
0924 09:56:14.146602 (+ 306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:56:14.146606 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:56:14.146663 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:56:14.146747 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 09:56:14.147283 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:14.147980 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:14.149687 (+ 1707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:14.150440 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:14.150552 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:14.150641 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0924 09:56:14.150946 (+ 305us) server_negotiation.cc:299] Negotiation successful
0924 09:56:14.151065 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260924 09:56:14.151427 4996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:14.145489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:38695 (local address 127.0.97.3:35703)
0924 09:56:14.145676 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0924 09:56:14.145682 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 09:56:14.146713 (+ 1031us) server_negotiation.cc:407] Connection header received
0924 09:56:14.146846 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:56:14.146852 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:56:14.146911 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:56:14.146982 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 09:56:14.147518 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:14.148196 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:14.149822 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:14.150577 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:14.150680 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:14.150772 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0924 09:56:14.151055 (+ 283us) server_negotiation.cc:299] Negotiation successful
0924 09:56:14.151171 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 09:56:14.151499 5008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:14.145313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:37345 (local address 127.0.97.1:34569)
0924 09:56:14.146206 (+ 893us) negotiation.cc:107] Waiting for socket to connect
0924 09:56:14.146224 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 09:56:14.146443 (+ 219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:56:14.146776 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:56:14.146780 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:56:14.146826 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 09:56:14.147125 (+ 299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:14.147134 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:14.148095 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:14.148100 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:14.149540 (+ 1440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:14.149551 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:14.150627 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:14.150633 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:14.150798 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:14.150830 (+ 32us) client_negotiation.cc:769] Sending connection context
0924 09:56:14.150905 (+ 75us) client_negotiation.cc:240] Negotiation successful
0924 09:56:14.151023 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":772,"mutex_wait_us":1,"thread_start_us":596,"threads_started":1}
I20260924 09:56:14.151523 5009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:14.145407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:35703 (local address 127.0.97.1:38695)
0924 09:56:14.146618 (+ 1211us) negotiation.cc:107] Waiting for socket to connect
0924 09:56:14.146631 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 09:56:14.146730 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:56:14.147011 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:56:14.147015 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:56:14.147037 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 09:56:14.147373 (+ 336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:14.147382 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:14.148320 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:14.148325 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:14.149678 (+ 1353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:14.149689 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:14.150778 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:14.150784 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:14.150921 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:14.150942 (+ 21us) client_negotiation.cc:769] Sending connection context
0924 09:56:14.151013 (+ 71us) client_negotiation.cc:240] Negotiation successful
0924 09:56:14.151129 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1115,"thread_start_us":919,"threads_started":1}
I20260924 09:56:14.152381 4936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f5f65ad44a64307b7a3b8c37cfd749f" candidate_uuid: "467506185daf40dc942de690005ce088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50fbe96f069f4ac2af2133a074ab8e5f" is_pre_election: true
I20260924 09:56:14.152390 4804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f5f65ad44a64307b7a3b8c37cfd749f" candidate_uuid: "467506185daf40dc942de690005ce088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29f47496f0844e8cb35eec7c187794e5" is_pre_election: true
I20260924 09:56:14.152805 4804 raft_consensus.cc:2454] T 1f5f65ad44a64307b7a3b8c37cfd749f P 29f47496f0844e8cb35eec7c187794e5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 467506185daf40dc942de690005ce088 in term 0.
I20260924 09:56:14.152828 4936 raft_consensus.cc:2454] T 1f5f65ad44a64307b7a3b8c37cfd749f P 50fbe96f069f4ac2af2133a074ab8e5f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 467506185daf40dc942de690005ce088 in term 0.
I20260924 09:56:14.153371 4606 leader_election.cc:302] T 1f5f65ad44a64307b7a3b8c37cfd749f P 467506185daf40dc942de690005ce088 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 467506185daf40dc942de690005ce088, 50fbe96f069f4ac2af2133a074ab8e5f; no voters:
I20260924 09:56:14.153705 5005 raft_consensus.cc:2781] T 1f5f65ad44a64307b7a3b8c37cfd749f P 467506185daf40dc942de690005ce088 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 09:56:14.153853 5005 raft_consensus.cc:482] T 1f5f65ad44a64307b7a3b8c37cfd749f P 467506185daf40dc942de690005ce088 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 09:56:14.153951 5005 raft_consensus.cc:3037] T 1f5f65ad44a64307b7a3b8c37cfd749f P 467506185daf40dc942de690005ce088 [term 0 FOLLOWER]: Advancing to term 1
I20260924 09:56:14.156015 5005 raft_consensus.cc:504] T 1f5f65ad44a64307b7a3b8c37cfd749f P 467506185daf40dc942de690005ce088 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f47496f0844e8cb35eec7c187794e5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37345 } } peers { permanent_uuid: "50fbe96f069f4ac2af2133a074ab8e5f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35703 } } peers { permanent_uuid: "467506185daf40dc942de690005ce088" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35417 } }
I20260924 09:56:14.156523 5005 leader_election.cc:288] T 1f5f65ad44a64307b7a3b8c37cfd749f P 467506185daf40dc942de690005ce088 [CANDIDATE]: Term 1 election: Requested vote from peers 29f47496f0844e8cb35eec7c187794e5 (127.0.97.2:37345), 50fbe96f069f4ac2af2133a074ab8e5f (127.0.97.3:35703)
I20260924 09:56:14.156930 4804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f5f65ad44a64307b7a3b8c37cfd749f" candidate_uuid: "467506185daf40dc942de690005ce088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29f47496f0844e8cb35eec7c187794e5"
I20260924 09:56:14.156965 4936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f5f65ad44a64307b7a3b8c37cfd749f" candidate_uuid: "467506185daf40dc942de690005ce088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50fbe96f069f4ac2af2133a074ab8e5f"
I20260924 09:56:14.157126 4804 raft_consensus.cc:3037] T 1f5f65ad44a64307b7a3b8c37cfd749f P 29f47496f0844e8cb35eec7c187794e5 [term 0 FOLLOWER]: Advancing to term 1
I20260924 09:56:14.157126 4936 raft_consensus.cc:3037] T 1f5f65ad44a64307b7a3b8c37cfd749f P 50fbe96f069f4ac2af2133a074ab8e5f [term 0 FOLLOWER]: Advancing to term 1
I20260924 09:56:14.160141 4804 raft_consensus.cc:2454] T 1f5f65ad44a64307b7a3b8c37cfd749f P 29f47496f0844e8cb35eec7c187794e5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 467506185daf40dc942de690005ce088 in term 1.
I20260924 09:56:14.160141 4936 raft_consensus.cc:2454] T 1f5f65ad44a64307b7a3b8c37cfd749f P 50fbe96f069f4ac2af2133a074ab8e5f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 467506185daf40dc942de690005ce088 in term 1.
I20260924 09:56:14.160530 4605 leader_election.cc:302] T 1f5f65ad44a64307b7a3b8c37cfd749f P 467506185daf40dc942de690005ce088 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 29f47496f0844e8cb35eec7c187794e5, 467506185daf40dc942de690005ce088, 50fbe96f069f4ac2af2133a074ab8e5f; no voters:
I20260924 09:56:14.160820 5005 raft_consensus.cc:2781] T 1f5f65ad44a64307b7a3b8c37cfd749f P 467506185daf40dc942de690005ce088 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 09:56:14.161201 5005 raft_consensus.cc:686] T 1f5f65ad44a64307b7a3b8c37cfd749f P 467506185daf40dc942de690005ce088 [term 1 LEADER]: Becoming Leader. State: Replica: 467506185daf40dc942de690005ce088, State: Running, Role: LEADER
I20260924 09:56:14.161644 5005 consensus_queue.cc:237] T 1f5f65ad44a64307b7a3b8c37cfd749f P 467506185daf40dc942de690005ce088 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f47496f0844e8cb35eec7c187794e5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37345 } } peers { permanent_uuid: "50fbe96f069f4ac2af2133a074ab8e5f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35703 } } peers { permanent_uuid: "467506185daf40dc942de690005ce088" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35417 } }
I20260924 09:56:14.165637 4528 catalog_manager.cc:5495] T 1f5f65ad44a64307b7a3b8c37cfd749f P 467506185daf40dc942de690005ce088 reported cstate change: term changed from 0 to 1, leader changed from <none> to 467506185daf40dc942de690005ce088 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "467506185daf40dc942de690005ce088" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f47496f0844e8cb35eec7c187794e5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37345 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "50fbe96f069f4ac2af2133a074ab8e5f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35703 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "467506185daf40dc942de690005ce088" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35417 } health_report { overall_health: HEALTHY } } }
I20260924 09:56:14.186281 4998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:14.181738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47324 (local address 127.0.97.1:35417)
0924 09:56:14.181922 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0924 09:56:14.181928 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 09:56:14.182026 (+ 98us) server_negotiation.cc:407] Connection header received
0924 09:56:14.182196 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:56:14.182200 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:56:14.182271 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:56:14.182381 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 09:56:14.182642 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:14.183353 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:14.184561 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:14.184943 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:14.185023 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:14.185388 (+ 365us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 09:56:14.185858 (+ 470us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 09:56:14.185956 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0924 09:56:14.185994 (+ 38us) server_negotiation.cc:299] Negotiation successful
0924 09:56:14.186051 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 09:56:14.194048 4936 raft_consensus.cc:1261] T 1f5f65ad44a64307b7a3b8c37cfd749f P 50fbe96f069f4ac2af2133a074ab8e5f [term 1 FOLLOWER]: Refusing update from remote peer 467506185daf40dc942de690005ce088: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 09:56:14.194048 4804 raft_consensus.cc:1261] T 1f5f65ad44a64307b7a3b8c37cfd749f P 29f47496f0844e8cb35eec7c187794e5 [term 1 FOLLOWER]: Refusing update from remote peer 467506185daf40dc942de690005ce088: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 09:56:14.195058 5010 consensus_queue.cc:1035] T 1f5f65ad44a64307b7a3b8c37cfd749f P 467506185daf40dc942de690005ce088 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29f47496f0844e8cb35eec7c187794e5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 09:56:14.195395 5005 consensus_queue.cc:1035] T 1f5f65ad44a64307b7a3b8c37cfd749f P 467506185daf40dc942de690005ce088 [LEADER]: Connected to new peer: Peer: permanent_uuid: "50fbe96f069f4ac2af2133a074ab8e5f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35703 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 09:56:14.206633 5013 mvcc.cc:205] Tried to move back new op lower bound from 7332838499088814080 to 7332838498969354240. Current Snapshot: MvccSnapshot[applied={T|T < 7332838499088814080}]
I20260924 09:56:14.207715 5014 mvcc.cc:205] Tried to move back new op lower bound from 7332838499088814080 to 7332838498969354240. Current Snapshot: MvccSnapshot[applied={T|T < 7332838499088814080}]
I20260924 09:56:14.219436 4528 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58660:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 09:56:14.219640 4528 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58660:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 09:56:14.222818 4528 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9c917eb636544e09ad3eec94c30189fd: Sending DeleteTablet for 3 replicas of tablet 1f5f65ad44a64307b7a3b8c37cfd749f
I20260924 09:56:14.223826 4784 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f5f65ad44a64307b7a3b8c37cfd749f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 09:56:14 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:48354
I20260924 09:56:14.224181 4651 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f5f65ad44a64307b7a3b8c37cfd749f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 09:56:14 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:47318
I20260924 09:56:14.224416 4916 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f5f65ad44a64307b7a3b8c37cfd749f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 09:56:14 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:59956
I20260924 09:56:14.224587 5022 tablet_replica.cc:330] T 1f5f65ad44a64307b7a3b8c37cfd749f P 29f47496f0844e8cb35eec7c187794e5: stopping tablet replica
I20260924 09:56:14.224959 5022 raft_consensus.cc:2229] T 1f5f65ad44a64307b7a3b8c37cfd749f P 29f47496f0844e8cb35eec7c187794e5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 09:56:14.225486 5022 raft_consensus.cc:2258] T 1f5f65ad44a64307b7a3b8c37cfd749f P 29f47496f0844e8cb35eec7c187794e5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 09:56:14.226203 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu pid 4589
I20260924 09:56:14.227614 5022 ts_tablet_manager.cc:1905] T 1f5f65ad44a64307b7a3b8c37cfd749f P 29f47496f0844e8cb35eec7c187794e5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 09:56:14.234509 5022 ts_tablet_manager.cc:1918] T 1f5f65ad44a64307b7a3b8c37cfd749f P 29f47496f0844e8cb35eec7c187794e5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 09:56:14.234754 5022 log.cc:1196] T 1f5f65ad44a64307b7a3b8c37cfd749f P 29f47496f0844e8cb35eec7c187794e5: Deleting WAL directory at /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790243735634835-388-0/minicluster-data/ts-1/wal/wals/1f5f65ad44a64307b7a3b8c37cfd749f
I20260924 09:56:14.235359 5022 ts_tablet_manager.cc:1939] T 1f5f65ad44a64307b7a3b8c37cfd749f P 29f47496f0844e8cb35eec7c187794e5: Deleting consensus metadata
I20260924 09:56:14.237258 4513 catalog_manager.cc:4841] TS 29f47496f0844e8cb35eec7c187794e5 (127.0.97.2:37345): tablet 1f5f65ad44a64307b7a3b8c37cfd749f (table test-table [id=6090a00cf22345f7b96ee30ee1b6ce34]) successfully deleted
I20260924 09:56:14.242465 5024 tablet_replica.cc:330] T 1f5f65ad44a64307b7a3b8c37cfd749f P 50fbe96f069f4ac2af2133a074ab8e5f: stopping tablet replica
I20260924 09:56:14.242830 5024 raft_consensus.cc:2229] T 1f5f65ad44a64307b7a3b8c37cfd749f P 50fbe96f069f4ac2af2133a074ab8e5f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 09:56:14.243227 5024 raft_consensus.cc:2258] T 1f5f65ad44a64307b7a3b8c37cfd749f P 50fbe96f069f4ac2af2133a074ab8e5f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 09:56:14.247155 5024 ts_tablet_manager.cc:1905] T 1f5f65ad44a64307b7a3b8c37cfd749f P 50fbe96f069f4ac2af2133a074ab8e5f: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 24 09:56:14 dist-test-slave-w8d2 krb5kdc[4484](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243772, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260924 09:56:14.252851 5023 tablet_replica.cc:330] T 1f5f65ad44a64307b7a3b8c37cfd749f P 467506185daf40dc942de690005ce088: stopping tablet replica
I20260924 09:56:14.253300 5023 raft_consensus.cc:2229] T 1f5f65ad44a64307b7a3b8c37cfd749f P 467506185daf40dc942de690005ce088 [term 1 LEADER]: Raft consensus shutting down.
I20260924 09:56:14.254052 5023 raft_consensus.cc:2258] T 1f5f65ad44a64307b7a3b8c37cfd749f P 467506185daf40dc942de690005ce088 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 09:56:14.256778 5023 ts_tablet_manager.cc:1905] T 1f5f65ad44a64307b7a3b8c37cfd749f P 467506185daf40dc942de690005ce088: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 09:56:14.256822 5024 ts_tablet_manager.cc:1918] T 1f5f65ad44a64307b7a3b8c37cfd749f P 50fbe96f069f4ac2af2133a074ab8e5f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 09:56:14.257052 5024 log.cc:1196] T 1f5f65ad44a64307b7a3b8c37cfd749f P 50fbe96f069f4ac2af2133a074ab8e5f: Deleting WAL directory at /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790243735634835-388-0/minicluster-data/ts-2/wal/wals/1f5f65ad44a64307b7a3b8c37cfd749f
I20260924 09:56:14.257596 5024 ts_tablet_manager.cc:1939] T 1f5f65ad44a64307b7a3b8c37cfd749f P 50fbe96f069f4ac2af2133a074ab8e5f: Deleting consensus metadata
I20260924 09:56:14.258499 4998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:14.228056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47334 (local address 127.0.97.1:35417)
0924 09:56:14.228235 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0924 09:56:14.228242 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 09:56:14.239827 (+ 11585us) server_negotiation.cc:407] Connection header received
0924 09:56:14.240098 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:56:14.240105 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:56:14.240193 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:56:14.240272 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:56:14.243579 (+ 3307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:14.244407 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:14.245871 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:14.246478 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:14.246597 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:14.251380 (+ 4783us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:56:14.251395 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:56:14.251412 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:56:14.251453 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:56:14.255194 (+ 3741us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:56:14.255978 (+ 784us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:56:14.255986 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:56:14.255990 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:56:14.256149 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:56:14.256574 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:56:14.256580 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:56:14.256583 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:56:14.257004 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:56:14.257316 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0924 09:56:14.257852 (+ 536us) server_negotiation.cc:299] Negotiation successful
0924 09:56:14.258088 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 09:56:14.259596 4514 catalog_manager.cc:4841] TS 50fbe96f069f4ac2af2133a074ab8e5f (127.0.97.3:35703): tablet 1f5f65ad44a64307b7a3b8c37cfd749f (table test-table [id=6090a00cf22345f7b96ee30ee1b6ce34]) successfully deleted
I20260924 09:56:14.260006 4711 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 09:56:14.356191 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu with pid 4589
W20260924 09:56:14.369957 4512 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:35417 (error 108)
I20260924 09:56:14.370630 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu pid 4722
I20260924 09:56:14.371626 4997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:14.371186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35417 (local address 127.0.0.1:47346)
0924 09:56:14.371406 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0924 09:56:14.371502 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:35417: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
W20260924 09:56:14.371832 4512 catalog_manager.cc:4584] TS 467506185daf40dc942de690005ce088 (127.0.97.1:35417): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1f5f65ad44a64307b7a3b8c37cfd749f: Network error: Client connection negotiation failed: client connection to 127.0.97.1:35417: connect: Connection refused (error 111)
Sep 24 09:56:14 dist-test-slave-w8d2 krb5kdc[4484](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243772, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260924 09:56:14.383960 4995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:14.371317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48364 (local address 127.0.97.2:37345)
0924 09:56:14.371485 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0924 09:56:14.371491 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 09:56:14.371596 (+ 105us) server_negotiation.cc:407] Connection header received
0924 09:56:14.371800 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:56:14.371806 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:56:14.371893 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:56:14.371981 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:56:14.373247 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:14.373961 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:14.375093 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:14.375634 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:14.375738 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:14.379208 (+ 3470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:56:14.379220 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:56:14.379267 (+ 47us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:56:14.379307 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:56:14.381570 (+ 2263us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:56:14.382124 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:56:14.382132 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:56:14.382134 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:56:14.382245 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:56:14.382561 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:56:14.382566 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:56:14.382569 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:56:14.382892 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:56:14.383154 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0924 09:56:14.383466 (+ 312us) server_negotiation.cc:299] Negotiation successful
0924 09:56:14.383677 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 09:56:14.384536 4844 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 09:56:14.425387 4997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:14.424880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35417 (local address 127.0.0.1:47350)
0924 09:56:14.425092 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0924 09:56:14.425204 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:35417: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260924 09:56:14.426321 4997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:14.425969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35417 (local address 127.0.0.1:47356)
0924 09:56:14.426097 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0924 09:56:14.426176 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:35417: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260924 09:56:14.470599 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu with pid 4722
I20260924 09:56:14.483454 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu pid 4854
I20260924 09:56:14.486999 4997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:14.486616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35417 (local address 127.0.0.1:47372)
0924 09:56:14.486777 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0924 09:56:14.486848 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:35417: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260924 09:56:14.487871 4997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:14.487589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35417 (local address 127.0.0.1:47382)
0924 09:56:14.487705 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0924 09:56:14.487779 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:35417: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
Sep 24 09:56:14 dist-test-slave-w8d2 krb5kdc[4484](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243772, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260924 09:56:14.497013 4996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:14.484122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59966 (local address 127.0.97.3:35703)
0924 09:56:14.484301 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0924 09:56:14.484306 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 09:56:14.484324 (+ 18us) server_negotiation.cc:407] Connection header received
0924 09:56:14.484486 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:56:14.484489 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:56:14.484549 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:56:14.484625 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:56:14.485763 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:14.486450 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:14.487948 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:14.488341 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:14.488440 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:14.492099 (+ 3659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:56:14.492113 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:56:14.492130 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:56:14.492165 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:56:14.494612 (+ 2447us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:56:14.495174 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:56:14.495181 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:56:14.495183 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:56:14.495341 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:56:14.495660 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:56:14.495666 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:56:14.495669 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:56:14.495996 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:56:14.496246 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0924 09:56:14.496541 (+ 295us) server_negotiation.cc:299] Negotiation successful
0924 09:56:14.496744 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260924 09:56:14.497593 4976 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 09:56:14.568511 4997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:14.568072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35417 (local address 127.0.0.1:47386)
0924 09:56:14.568267 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0924 09:56:14.568362 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:35417: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260924 09:56:14.569485 4997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:14.569145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35417 (local address 127.0.0.1:47392)
0924 09:56:14.569302 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0924 09:56:14.569378 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:35417: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260924 09:56:14.583437 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu with pid 4854
I20260924 09:56:14.595947 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu pid 4497
I20260924 09:56:14.596975 4558 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 09:56:14.704279 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu with pid 4497
2026-09-24T09:56:14Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (3795 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-24T09:56:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T09:56:14Z Disabled control of system clock
I20260924 09:56:14.769240 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
/tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790243735634835-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790243735634835-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790243735634835-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790243735634835-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33771
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:41705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:33771
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260924 09:56:15.067749 5034 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 09:56:15.068029 5034 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 09:56:15.068111 5034 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 09:56:15.075124 5034 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 09:56:15.075255 5034 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 09:56:15.075318 5034 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 09:56:15.075380 5034 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 09:56:15.075428 5034 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 09:56:15.084502 5034 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790243735634835-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790243735634835-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:33771
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33771
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790243735634835-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790243735634835-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 09:56:15.085732 5034 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 09:56:15.087419 5034 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 09:56:15.097942 5039 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:56:15.097981 5040 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:56:15.098012 5042 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 09:56:15.098893 5034 server_base.cc:1034] running on GCE node
I20260924 09:56:15.099641 5034 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 09:56:15.100865 5034 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 09:56:15.102054 5034 hybrid_clock.cc:648] HybridClock initialized: now 1790243775102006 us; error 45 us; skew 500 ppm
I20260924 09:56:15.102360 5034 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 09:56:15.104413 5034 webserver.cc:460] Webserver started at http://127.0.97.62:43631/ using document root <none> and password file <none>
I20260924 09:56:15.105046 5034 fs_manager.cc:359] Metadata directory not provided
I20260924 09:56:15.105132 5034 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 09:56:15.105435 5034 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 09:56:15.108191 5034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790243735634835-388-0/minicluster-data/master-0/data/instance:
uuid: "6e2ddf92c18c49e2bc730ec50830f8c6"
format_stamp: "Formatted at 2026-09-24 09:56:15 on dist-test-slave-w8d2"
server_key: "b50319e89afc8347748fbc9210c023ee"
server_key_iv: "f8f06b677ccc9c002311a7793c7aaf21"
server_key_version: "encryptionkey@0"
I20260924 09:56:15.108913 5034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790243735634835-388-0/minicluster-data/master-0/wal/instance:
uuid: "6e2ddf92c18c49e2bc730ec50830f8c6"
format_stamp: "Formatted at 2026-09-24 09:56:15 on dist-test-slave-w8d2"
server_key: "b50319e89afc8347748fbc9210c023ee"
server_key_iv: "f8f06b677ccc9c002311a7793c7aaf21"
server_key_version: "encryptionkey@0"
I20260924 09:56:15.113679 5034 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 09:56:15.116806 5048 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 09:56:15.117983 5034 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 09:56:15.118121 5034 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790243735634835-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790243735634835-388-0/minicluster-data/master-0/wal
uuid: "6e2ddf92c18c49e2bc730ec50830f8c6"
format_stamp: "Formatted at 2026-09-24 09:56:15 on dist-test-slave-w8d2"
server_key: "b50319e89afc8347748fbc9210c023ee"
server_key_iv: "f8f06b677ccc9c002311a7793c7aaf21"
server_key_version: "encryptionkey@0"
I20260924 09:56:15.118299 5034 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790243735634835-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790243735634835-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790243735634835-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 09:56:15.123632 5034 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 09:56:15.124300 5034 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 09:56:15.124557 5034 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 09:56:15.140928 5034 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:33771
I20260924 09:56:15.140945 5099 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:33771 every 8 connection(s)
I20260924 09:56:15.142246 5034 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790243735634835-388-0/minicluster-data/master-0/data/info.pb
I20260924 09:56:15.145691 5100 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 09:56:15.152344 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu as pid 5034
I20260924 09:56:15.152546 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790243735634835-388-0/minicluster-data/master-0/wal/instance
I20260924 09:56:15.153091 388 external_mini_cluster.cc:1428] Setting key 9f2933c2b0d6a96d5ea596b83aea09c4
I20260924 09:56:15.155331 5100 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e2ddf92c18c49e2bc730ec50830f8c6: Bootstrap starting.
I20260924 09:56:15.158311 5100 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6e2ddf92c18c49e2bc730ec50830f8c6: Neither blocks nor log segments found. Creating new log.
I20260924 09:56:15.159536 5100 log.cc:824] T 00000000000000000000000000000000 P 6e2ddf92c18c49e2bc730ec50830f8c6: Log is configured to *not* fsync() on all Append() calls
I20260924 09:56:15.160830 5103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:15.154914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44102 (local address 127.0.97.62:33771)
0924 09:56:15.155833 (+ 919us) server_negotiation.cc:206] Beginning negotiation
0924 09:56:15.155844 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 09:56:15.155883 (+ 39us) server_negotiation.cc:407] Connection header received
0924 09:56:15.157464 (+ 1581us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:56:15.157485 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:56:15.157860 (+ 375us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:56:15.158168 (+ 308us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:56:15.159135 (+ 967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:56:15.159141 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:56:15.159153 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 09:56:15.159166 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:56:15.159252 (+ 86us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 09:56:15.159275 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:56:15.159411 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0924 09:56:15.159523 (+ 112us) server_negotiation.cc:299] Negotiation successful
0924 09:56:15.159740 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":500,"thread_start_us":296,"threads_started":1}
I20260924 09:56:15.162158 5100 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e2ddf92c18c49e2bc730ec50830f8c6: No bootstrap required, opened a new log
I20260924 09:56:15.169062 5100 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6e2ddf92c18c49e2bc730ec50830f8c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e2ddf92c18c49e2bc730ec50830f8c6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33771 } }
I20260924 09:56:15.169360 5100 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6e2ddf92c18c49e2bc730ec50830f8c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 09:56:15.169447 5100 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6e2ddf92c18c49e2bc730ec50830f8c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e2ddf92c18c49e2bc730ec50830f8c6, State: Initialized, Role: FOLLOWER
I20260924 09:56:15.170017 5100 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6e2ddf92c18c49e2bc730ec50830f8c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e2ddf92c18c49e2bc730ec50830f8c6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33771 } }
I20260924 09:56:15.170274 5100 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6e2ddf92c18c49e2bc730ec50830f8c6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 09:56:15.170392 5100 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6e2ddf92c18c49e2bc730ec50830f8c6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 09:56:15.170529 5100 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6e2ddf92c18c49e2bc730ec50830f8c6 [term 0 FOLLOWER]: Advancing to term 1
I20260924 09:56:15.172555 5100 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6e2ddf92c18c49e2bc730ec50830f8c6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e2ddf92c18c49e2bc730ec50830f8c6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33771 } }
I20260924 09:56:15.172981 5100 leader_election.cc:302] T 00000000000000000000000000000000 P 6e2ddf92c18c49e2bc730ec50830f8c6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6e2ddf92c18c49e2bc730ec50830f8c6; no voters:
I20260924 09:56:15.173491 5100 leader_election.cc:288] T 00000000000000000000000000000000 P 6e2ddf92c18c49e2bc730ec50830f8c6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 09:56:15.173597 5105 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6e2ddf92c18c49e2bc730ec50830f8c6 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 09:56:15.174211 5105 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6e2ddf92c18c49e2bc730ec50830f8c6 [term 1 LEADER]: Becoming Leader. State: Replica: 6e2ddf92c18c49e2bc730ec50830f8c6, State: Running, Role: LEADER
I20260924 09:56:15.174683 5105 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6e2ddf92c18c49e2bc730ec50830f8c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e2ddf92c18c49e2bc730ec50830f8c6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33771 } }
I20260924 09:56:15.175314 5100 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6e2ddf92c18c49e2bc730ec50830f8c6 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 09:56:15.177184 5107 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e2ddf92c18c49e2bc730ec50830f8c6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6e2ddf92c18c49e2bc730ec50830f8c6. Latest consensus state: current_term: 1 leader_uuid: "6e2ddf92c18c49e2bc730ec50830f8c6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e2ddf92c18c49e2bc730ec50830f8c6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33771 } } }
I20260924 09:56:15.177529 5107 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e2ddf92c18c49e2bc730ec50830f8c6 [sys.catalog]: This master's current role is: LEADER
I20260924 09:56:15.177997 5106 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e2ddf92c18c49e2bc730ec50830f8c6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6e2ddf92c18c49e2bc730ec50830f8c6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e2ddf92c18c49e2bc730ec50830f8c6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33771 } } }
I20260924 09:56:15.178293 5106 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e2ddf92c18c49e2bc730ec50830f8c6 [sys.catalog]: This master's current role is: LEADER
I20260924 09:56:15.178503 5114 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 09:56:15.183519 5114 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 09:56:15.191051 5114 catalog_manager.cc:1347] Generated new cluster ID: 421f6aac501f450c96e117d9056000eb
I20260924 09:56:15.191150 5114 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 09:56:15.215204 5114 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 09:56:15.215651 5114 catalog_manager.cc:1504] Loading token signing keys...
I20260924 09:56:15.226383 5114 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6e2ddf92c18c49e2bc730ec50830f8c6: Generated new TSK 0
I20260924 09:56:15.227420 5114 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 09:56:15.241057 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
/tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790243735634835-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790243735634835-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790243735634835-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790243735634835-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33771
--builtin_ntp_servers=127.0.97.20:41705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260924 09:56:15.545159 5124 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 09:56:15.545424 5124 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 09:56:15.545540 5124 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 09:56:15.552498 5124 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 09:56:15.552598 5124 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 09:56:15.552711 5124 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260924 09:56:15.561477 5124 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790243735634835-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790243735634835-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790243735634835-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33771
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790243735634835-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 09:56:15.562657 5124 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 09:56:15.564392 5124 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 09:56:15.576471 5129 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:56:15.576495 5130 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:56:15.576543 5132 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 09:56:15.577342 5124 server_base.cc:1034] running on GCE node
I20260924 09:56:15.577914 5124 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 09:56:15.578876 5124 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 09:56:15.580109 5124 hybrid_clock.cc:648] HybridClock initialized: now 1790243775580059 us; error 38 us; skew 500 ppm
I20260924 09:56:15.580430 5124 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 09:56:15.582567 5124 webserver.cc:460] Webserver started at http://127.0.97.1:40981/ using document root <none> and password file <none>
I20260924 09:56:15.583199 5124 fs_manager.cc:359] Metadata directory not provided
I20260924 09:56:15.583333 5124 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 09:56:15.583635 5124 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 09:56:15.586390 5124 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790243735634835-388-0/minicluster-data/ts-0/data/instance:
uuid: "bd85c5a9f1c54df4b1efa5beef5b34b6"
format_stamp: "Formatted at 2026-09-24 09:56:15 on dist-test-slave-w8d2"
server_key: "ec06f68c3194dce3e58e6a8ba54cf725"
server_key_iv: "205298febff856579e04d65af28aaebf"
server_key_version: "encryptionkey@0"
I20260924 09:56:15.587107 5124 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790243735634835-388-0/minicluster-data/ts-0/wal/instance:
uuid: "bd85c5a9f1c54df4b1efa5beef5b34b6"
format_stamp: "Formatted at 2026-09-24 09:56:15 on dist-test-slave-w8d2"
server_key: "ec06f68c3194dce3e58e6a8ba54cf725"
server_key_iv: "205298febff856579e04d65af28aaebf"
server_key_version: "encryptionkey@0"
I20260924 09:56:15.591871 5124 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260924 09:56:15.595067 5138 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 09:56:15.596367 5124 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 09:56:15.596515 5124 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790243735634835-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790243735634835-388-0/minicluster-data/ts-0/wal
uuid: "bd85c5a9f1c54df4b1efa5beef5b34b6"
format_stamp: "Formatted at 2026-09-24 09:56:15 on dist-test-slave-w8d2"
server_key: "ec06f68c3194dce3e58e6a8ba54cf725"
server_key_iv: "205298febff856579e04d65af28aaebf"
server_key_version: "encryptionkey@0"
I20260924 09:56:15.596705 5124 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790243735634835-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790243735634835-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790243735634835-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 09:56:15.601900 5124 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 09:56:15.602659 5124 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 09:56:15.602886 5124 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 09:56:15.603787 5124 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 09:56:15.605075 5124 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 09:56:15.605159 5124 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 09:56:15.605257 5124 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 09:56:15.605325 5124 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 09:56:15.632010 5124 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:43047
I20260924 09:56:15.632050 5250 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:43047 every 8 connection(s)
I20260924 09:56:15.633225 5124 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790243735634835-388-0/minicluster-data/ts-0/data/info.pb
I20260924 09:56:15.637130 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu as pid 5124
I20260924 09:56:15.637315 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790243735634835-388-0/minicluster-data/ts-0/wal/instance
I20260924 09:56:15.637871 388 external_mini_cluster.cc:1428] Setting key c62cdca61bbef6c9cfa440a18f66dd0f
I20260924 09:56:15.640532 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
/tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790243735634835-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790243735634835-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790243735634835-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790243735634835-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33771
--builtin_ntp_servers=127.0.97.20:41705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260924 09:56:15.640705 5103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:15.635543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:52951 (local address 127.0.97.62:33771)
0924 09:56:15.635694 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0924 09:56:15.635698 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 09:56:15.636957 (+ 1259us) server_negotiation.cc:407] Connection header received
0924 09:56:15.638647 (+ 1690us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:56:15.638651 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:56:15.638700 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:56:15.638798 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:56:15.640059 (+ 1261us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:56:15.640064 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:56:15.640068 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 09:56:15.640087 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:56:15.640125 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 09:56:15.640139 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:56:15.640258 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0924 09:56:15.640387 (+ 129us) server_negotiation.cc:299] Negotiation successful
0924 09:56:15.640446 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 09:56:15.641546 5253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:15.635973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33771 (local address 127.0.97.1:52951)
0924 09:56:15.636812 (+ 839us) negotiation.cc:107] Waiting for socket to connect
0924 09:56:15.636847 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 09:56:15.638380 (+ 1533us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:56:15.638982 (+ 602us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:56:15.638990 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:56:15.639081 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0924 09:56:15.639786 (+ 705us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 09:56:15.639789 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 09:56:15.639824 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 09:56:15.639905 (+ 81us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 09:56:15.640223 (+ 318us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 09:56:15.640228 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 09:56:15.640233 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 09:56:15.640425 (+ 192us) client_negotiation.cc:240] Negotiation successful
0924 09:56:15.640612 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":233,"threads_started":1}
I20260924 09:56:15.642822 5251 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33771
I20260924 09:56:15.643086 5251 heartbeater.cc:464] Registering TS with master...
I20260924 09:56:15.643636 5251 heartbeater.cc:511] Master 127.0.97.62:33771 requested a full tablet report, sending...
I20260924 09:56:15.645442 5065 ts_manager.cc:194] Registered new tserver with Master: bd85c5a9f1c54df4b1efa5beef5b34b6 (127.0.97.1:43047)
W20260924 09:56:15.941898 5254 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 09:56:15.942170 5254 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 09:56:15.942276 5254 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 09:56:15.949368 5254 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 09:56:15.949462 5254 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 09:56:15.949600 5254 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260924 09:56:15.958488 5254 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790243735634835-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790243735634835-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790243735634835-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33771
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790243735634835-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 09:56:15.959781 5254 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 09:56:15.961454 5254 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 09:56:15.973785 5262 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:56:15.973850 5259 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:56:15.973822 5260 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 09:56:15.974442 5254 server_base.cc:1034] running on GCE node
I20260924 09:56:15.975014 5254 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 09:56:15.975961 5254 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 09:56:15.977150 5254 hybrid_clock.cc:648] HybridClock initialized: now 1790243775977108 us; error 40 us; skew 500 ppm
I20260924 09:56:15.977435 5254 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 09:56:15.982774 5254 webserver.cc:460] Webserver started at http://127.0.97.2:39071/ using document root <none> and password file <none>
I20260924 09:56:15.983497 5254 fs_manager.cc:359] Metadata directory not provided
I20260924 09:56:15.983594 5254 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 09:56:15.983878 5254 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 09:56:15.986568 5254 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790243735634835-388-0/minicluster-data/ts-1/data/instance:
uuid: "89abbf11becb47b8a69d4b743fbabd47"
format_stamp: "Formatted at 2026-09-24 09:56:15 on dist-test-slave-w8d2"
server_key: "8078eabc9b750da8c45db062a6371579"
server_key_iv: "d27135b467ed83cc98c9d1fdf41c3426"
server_key_version: "encryptionkey@0"
I20260924 09:56:15.987337 5254 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790243735634835-388-0/minicluster-data/ts-1/wal/instance:
uuid: "89abbf11becb47b8a69d4b743fbabd47"
format_stamp: "Formatted at 2026-09-24 09:56:15 on dist-test-slave-w8d2"
server_key: "8078eabc9b750da8c45db062a6371579"
server_key_iv: "d27135b467ed83cc98c9d1fdf41c3426"
server_key_version: "encryptionkey@0"
I20260924 09:56:15.992098 5254 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 09:56:15.995158 5268 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 09:56:15.996402 5254 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 09:56:15.996541 5254 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790243735634835-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790243735634835-388-0/minicluster-data/ts-1/wal
uuid: "89abbf11becb47b8a69d4b743fbabd47"
format_stamp: "Formatted at 2026-09-24 09:56:15 on dist-test-slave-w8d2"
server_key: "8078eabc9b750da8c45db062a6371579"
server_key_iv: "d27135b467ed83cc98c9d1fdf41c3426"
server_key_version: "encryptionkey@0"
I20260924 09:56:15.996737 5254 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790243735634835-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790243735634835-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790243735634835-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 09:56:16.002002 5254 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 09:56:16.002702 5254 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 09:56:16.002933 5254 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 09:56:16.003808 5254 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 09:56:16.005070 5254 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 09:56:16.005151 5254 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 09:56:16.005249 5254 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 09:56:16.005318 5254 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 09:56:16.033272 5254 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:39459
I20260924 09:56:16.033317 5380 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:39459 every 8 connection(s)
I20260924 09:56:16.034523 5254 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790243735634835-388-0/minicluster-data/ts-1/data/info.pb
I20260924 09:56:16.037949 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu as pid 5254
I20260924 09:56:16.038144 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790243735634835-388-0/minicluster-data/ts-1/wal/instance
I20260924 09:56:16.038645 388 external_mini_cluster.cc:1428] Setting key aa52c096b15f2782ee779a488c1d3f53
I20260924 09:56:16.041426 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
/tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790243735634835-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790243735634835-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790243735634835-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790243735634835-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33771
--builtin_ntp_servers=127.0.97.20:41705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260924 09:56:16.042572 5103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:16.036813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:36767 (local address 127.0.97.62:33771)
0924 09:56:16.036961 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0924 09:56:16.036966 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 09:56:16.038483 (+ 1517us) server_negotiation.cc:407] Connection header received
0924 09:56:16.040477 (+ 1994us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:56:16.040483 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:56:16.040532 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:56:16.040613 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:56:16.041986 (+ 1373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:56:16.041990 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:56:16.041993 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 09:56:16.042008 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:56:16.042037 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 09:56:16.042048 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:56:16.042142 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0924 09:56:16.042324 (+ 182us) server_negotiation.cc:299] Negotiation successful
0924 09:56:16.042369 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 09:56:16.043771 5383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:16.037279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33771 (local address 127.0.97.2:36767)
0924 09:56:16.038312 (+ 1033us) negotiation.cc:107] Waiting for socket to connect
0924 09:56:16.038357 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 09:56:16.040210 (+ 1853us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:56:16.040862 (+ 652us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:56:16.040873 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:56:16.040990 (+ 117us) client_negotiation.cc:189] Negotiated authn=SASL
0924 09:56:16.041678 (+ 688us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 09:56:16.041681 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 09:56:16.041726 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 09:56:16.041823 (+ 97us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 09:56:16.042130 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 09:56:16.042139 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 09:56:16.042147 (+ 8us) client_negotiation.cc:769] Sending connection context
0924 09:56:16.042398 (+ 251us) client_negotiation.cc:240] Negotiation successful
0924 09:56:16.042661 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":270,"threads_started":1}
I20260924 09:56:16.044967 5381 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33771
I20260924 09:56:16.045248 5381 heartbeater.cc:464] Registering TS with master...
I20260924 09:56:16.045856 5381 heartbeater.cc:511] Master 127.0.97.62:33771 requested a full tablet report, sending...
I20260924 09:56:16.047214 5065 ts_manager.cc:194] Registered new tserver with Master: 89abbf11becb47b8a69d4b743fbabd47 (127.0.97.2:39459)
W20260924 09:56:16.344833 5384 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 09:56:16.345131 5384 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 09:56:16.345257 5384 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 09:56:16.352439 5384 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 09:56:16.352535 5384 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 09:56:16.352694 5384 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260924 09:56:16.361567 5384 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790243735634835-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790243735634835-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790243735634835-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33771
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790243735634835-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 09:56:16.362752 5384 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 09:56:16.364442 5384 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 09:56:16.378556 5389 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:56:16.378567 5392 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:56:16.378899 5390 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 09:56:16.379311 5384 server_base.cc:1034] running on GCE node
I20260924 09:56:16.379932 5384 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 09:56:16.380820 5384 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 09:56:16.382009 5384 hybrid_clock.cc:648] HybridClock initialized: now 1790243776381962 us; error 47 us; skew 500 ppm
I20260924 09:56:16.382304 5384 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 09:56:16.384378 5384 webserver.cc:460] Webserver started at http://127.0.97.3:44743/ using document root <none> and password file <none>
I20260924 09:56:16.385106 5384 fs_manager.cc:359] Metadata directory not provided
I20260924 09:56:16.385205 5384 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 09:56:16.385485 5384 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 09:56:16.388243 5384 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790243735634835-388-0/minicluster-data/ts-2/data/instance:
uuid: "48954e80951c451e8d16b2171fb479b0"
format_stamp: "Formatted at 2026-09-24 09:56:16 on dist-test-slave-w8d2"
server_key: "64a556bbea0fa0041436a9d3f793254a"
server_key_iv: "290d8f7e5a1efbaed40488058c59ad91"
server_key_version: "encryptionkey@0"
I20260924 09:56:16.389014 5384 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790243735634835-388-0/minicluster-data/ts-2/wal/instance:
uuid: "48954e80951c451e8d16b2171fb479b0"
format_stamp: "Formatted at 2026-09-24 09:56:16 on dist-test-slave-w8d2"
server_key: "64a556bbea0fa0041436a9d3f793254a"
server_key_iv: "290d8f7e5a1efbaed40488058c59ad91"
server_key_version: "encryptionkey@0"
I20260924 09:56:16.393657 5384 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 09:56:16.396836 5398 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 09:56:16.398080 5384 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 09:56:16.398221 5384 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790243735634835-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790243735634835-388-0/minicluster-data/ts-2/wal
uuid: "48954e80951c451e8d16b2171fb479b0"
format_stamp: "Formatted at 2026-09-24 09:56:16 on dist-test-slave-w8d2"
server_key: "64a556bbea0fa0041436a9d3f793254a"
server_key_iv: "290d8f7e5a1efbaed40488058c59ad91"
server_key_version: "encryptionkey@0"
I20260924 09:56:16.398413 5384 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790243735634835-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790243735634835-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790243735634835-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 09:56:16.403750 5384 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 09:56:16.404464 5384 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 09:56:16.404690 5384 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 09:56:16.405515 5384 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 09:56:16.406757 5384 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 09:56:16.406844 5384 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 09:56:16.406940 5384 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 09:56:16.407006 5384 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 09:56:16.435379 5384 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:44147
I20260924 09:56:16.435406 5510 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:44147 every 8 connection(s)
I20260924 09:56:16.436668 5384 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790243735634835-388-0/minicluster-data/ts-2/data/info.pb
I20260924 09:56:16.439749 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu as pid 5384
I20260924 09:56:16.439929 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790243735634835-388-0/minicluster-data/ts-2/wal/instance
I20260924 09:56:16.440477 388 external_mini_cluster.cc:1428] Setting key 4e8f7c91c0258a2e3e1c83f9ddb90f60
I20260924 09:56:16.444588 5103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:16.438957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:47899 (local address 127.0.97.62:33771)
0924 09:56:16.439123 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0924 09:56:16.439128 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 09:56:16.440666 (+ 1538us) server_negotiation.cc:407] Connection header received
0924 09:56:16.442674 (+ 2008us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:56:16.442679 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:56:16.442739 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:56:16.442846 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:56:16.444077 (+ 1231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:56:16.444081 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:56:16.444085 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 09:56:16.444100 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:56:16.444129 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 09:56:16.444140 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:56:16.444227 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0924 09:56:16.444359 (+ 132us) server_negotiation.cc:299] Negotiation successful
0924 09:56:16.444402 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 09:56:16.445467 5513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:16.439501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33771 (local address 127.0.97.3:47899)
0924 09:56:16.440502 (+ 1001us) negotiation.cc:107] Waiting for socket to connect
0924 09:56:16.440554 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0924 09:56:16.442433 (+ 1879us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:56:16.443018 (+ 585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:56:16.443029 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:56:16.443145 (+ 116us) client_negotiation.cc:189] Negotiated authn=SASL
0924 09:56:16.443803 (+ 658us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 09:56:16.443806 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 09:56:16.443841 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 09:56:16.443923 (+ 82us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 09:56:16.444214 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 09:56:16.444220 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 09:56:16.444225 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 09:56:16.444410 (+ 185us) client_negotiation.cc:240] Negotiation successful
0924 09:56:16.444591 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":265,"threads_started":1}
I20260924 09:56:16.446602 5511 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33771
I20260924 09:56:16.446851 5511 heartbeater.cc:464] Registering TS with master...
I20260924 09:56:16.447391 5511 heartbeater.cc:511] Master 127.0.97.62:33771 requested a full tablet report, sending...
I20260924 09:56:16.448601 5065 ts_manager.cc:194] Registered new tserver with Master: 48954e80951c451e8d16b2171fb479b0 (127.0.97.3:44147)
I20260924 09:56:16.455113 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 09:56:16.461238 5103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:16.459173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44112 (local address 127.0.97.62:33771)
0924 09:56:16.459358 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0924 09:56:16.459364 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 09:56:16.459779 (+ 415us) server_negotiation.cc:407] Connection header received
0924 09:56:16.459886 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:56:16.459890 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:56:16.459935 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:56:16.460007 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:56:16.460758 (+ 751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:56:16.460762 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:56:16.460765 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 09:56:16.460775 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:56:16.460797 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 09:56:16.460806 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:56:16.460882 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 09:56:16.460941 (+ 59us) server_negotiation.cc:299] Negotiation successful
0924 09:56:16.460981 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 09:56:16.464556 5065 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44112:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 09:56:16.470911 5065 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 09:56:16.503072 5524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:16.482091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:39459 (local address 127.0.0.1:47434)
0924 09:56:16.494616 (+ 12525us) negotiation.cc:107] Waiting for socket to connect
0924 09:56:16.494633 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 09:56:16.494725 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:56:16.500701 (+ 5976us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:56:16.500708 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:56:16.500735 (+ 27us) client_negotiation.cc:189] Negotiated authn=SASL
0924 09:56:16.501537 (+ 802us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 09:56:16.501541 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 09:56:16.501569 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 09:56:16.501629 (+ 60us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 09:56:16.502112 (+ 483us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 09:56:16.502123 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 09:56:16.502132 (+ 9us) client_negotiation.cc:769] Sending connection context
0924 09:56:16.502290 (+ 158us) client_negotiation.cc:240] Negotiation successful
0924 09:56:16.502518 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12405,"thread_start_us":12699,"threads_started":1}
I20260924 09:56:16.503762 5523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:16.481859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:44147 (local address 127.0.0.1:34314)
0924 09:56:16.493818 (+ 11959us) negotiation.cc:107] Waiting for socket to connect
0924 09:56:16.493883 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0924 09:56:16.494054 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:56:16.499117 (+ 5063us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:56:16.499125 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:56:16.499226 (+ 101us) client_negotiation.cc:189] Negotiated authn=SASL
0924 09:56:16.501535 (+ 2309us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 09:56:16.501537 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 09:56:16.501561 (+ 24us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 09:56:16.501627 (+ 66us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 09:56:16.502857 (+ 1230us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 09:56:16.502861 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 09:56:16.502865 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 09:56:16.503008 (+ 143us) client_negotiation.cc:240] Negotiation successful
0924 09:56:16.503064 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11639,"thread_start_us":12345,"threads_started":1}
I20260924 09:56:16.503865 5525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:16.482092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34314 (local address 127.0.97.3:44147)
0924 09:56:16.497373 (+ 15281us) server_negotiation.cc:206] Beginning negotiation
0924 09:56:16.497383 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 09:56:16.497422 (+ 39us) server_negotiation.cc:407] Connection header received
0924 09:56:16.497509 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:56:16.497518 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:56:16.497798 (+ 280us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:56:16.497968 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:56:16.502047 (+ 4079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:56:16.502056 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:56:16.502070 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 09:56:16.502089 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:56:16.502148 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 09:56:16.502170 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:56:16.503319 (+ 1149us) server_negotiation.cc:1035] Waiting for connection context
0924 09:56:16.503388 (+ 69us) server_negotiation.cc:299] Negotiation successful
0924 09:56:16.503482 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15077,"thread_start_us":19642,"threads_started":1}
I20260924 09:56:16.504104 5526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:16.482456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47434 (local address 127.0.97.2:39459)
0924 09:56:16.500151 (+ 17695us) server_negotiation.cc:206] Beginning negotiation
0924 09:56:16.500159 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 09:56:16.500195 (+ 36us) server_negotiation.cc:407] Connection header received
0924 09:56:16.500252 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:56:16.500259 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:56:16.500488 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:56:16.500619 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:56:16.501864 (+ 1245us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:56:16.501875 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:56:16.501893 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 09:56:16.501921 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:56:16.501985 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 09:56:16.502005 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:56:16.502857 (+ 852us) server_negotiation.cc:1035] Waiting for connection context
0924 09:56:16.502929 (+ 72us) server_negotiation.cc:299] Negotiation successful
0924 09:56:16.503025 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17462,"thread_start_us":17343,"threads_started":1}
I20260924 09:56:16.507164 5527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:16.495603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44120 (local address 127.0.97.1:43047)
0924 09:56:16.499712 (+ 4109us) server_negotiation.cc:206] Beginning negotiation
0924 09:56:16.499720 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 09:56:16.502702 (+ 2982us) server_negotiation.cc:407] Connection header received
0924 09:56:16.502846 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:56:16.502855 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:56:16.503080 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:56:16.503224 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:56:16.505557 (+ 2333us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:56:16.505567 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:56:16.505580 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 09:56:16.505598 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:56:16.505658 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 09:56:16.505681 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:56:16.505800 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0924 09:56:16.506709 (+ 909us) server_negotiation.cc:299] Negotiation successful
0924 09:56:16.506887 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3955,"thread_start_us":3539,"threads_started":1}
I20260924 09:56:16.507625 5528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:16.496171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43047 (local address 127.0.0.1:44120)
0924 09:56:16.502604 (+ 6433us) negotiation.cc:107] Waiting for socket to connect
0924 09:56:16.502619 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 09:56:16.502716 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:56:16.504111 (+ 1395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:56:16.504116 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:56:16.504138 (+ 22us) client_negotiation.cc:189] Negotiated authn=SASL
0924 09:56:16.505311 (+ 1173us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 09:56:16.505315 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 09:56:16.505333 (+ 18us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 09:56:16.505378 (+ 45us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 09:56:16.506059 (+ 681us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 09:56:16.506064 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 09:56:16.506069 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 09:56:16.506196 (+ 127us) client_negotiation.cc:240] Negotiation successful
0924 09:56:16.506243 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6317,"thread_start_us":6175,"threads_started":1}
I20260924 09:56:16.507513 5446 tablet_service.cc:1467] Processing CreateTablet for tablet 92bd23242c07444692da2ec596bbc066 (DEFAULT_TABLE table=test-table [id=9ca7b23a01194200b2192105e8d1c22d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 09:56:16.507906 5316 tablet_service.cc:1467] Processing CreateTablet for tablet 92bd23242c07444692da2ec596bbc066 (DEFAULT_TABLE table=test-table [id=9ca7b23a01194200b2192105e8d1c22d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 09:56:16.508754 5446 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92bd23242c07444692da2ec596bbc066. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 09:56:16.508910 5316 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92bd23242c07444692da2ec596bbc066. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 09:56:16.513919 5186 tablet_service.cc:1467] Processing CreateTablet for tablet 92bd23242c07444692da2ec596bbc066 (DEFAULT_TABLE table=test-table [id=9ca7b23a01194200b2192105e8d1c22d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 09:56:16.515062 5186 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92bd23242c07444692da2ec596bbc066. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 09:56:16.519660 5529 tablet_bootstrap.cc:492] T 92bd23242c07444692da2ec596bbc066 P 89abbf11becb47b8a69d4b743fbabd47: Bootstrap starting.
I20260924 09:56:16.522773 5529 tablet_bootstrap.cc:654] T 92bd23242c07444692da2ec596bbc066 P 89abbf11becb47b8a69d4b743fbabd47: Neither blocks nor log segments found. Creating new log.
I20260924 09:56:16.524451 5529 log.cc:824] T 92bd23242c07444692da2ec596bbc066 P 89abbf11becb47b8a69d4b743fbabd47: Log is configured to *not* fsync() on all Append() calls
I20260924 09:56:16.524606 5530 tablet_bootstrap.cc:492] T 92bd23242c07444692da2ec596bbc066 P 48954e80951c451e8d16b2171fb479b0: Bootstrap starting.
I20260924 09:56:16.527662 5529 tablet_bootstrap.cc:492] T 92bd23242c07444692da2ec596bbc066 P 89abbf11becb47b8a69d4b743fbabd47: No bootstrap required, opened a new log
I20260924 09:56:16.527726 5532 tablet_bootstrap.cc:492] T 92bd23242c07444692da2ec596bbc066 P bd85c5a9f1c54df4b1efa5beef5b34b6: Bootstrap starting.
I20260924 09:56:16.527763 5530 tablet_bootstrap.cc:654] T 92bd23242c07444692da2ec596bbc066 P 48954e80951c451e8d16b2171fb479b0: Neither blocks nor log segments found. Creating new log.
I20260924 09:56:16.528038 5529 ts_tablet_manager.cc:1397] T 92bd23242c07444692da2ec596bbc066 P 89abbf11becb47b8a69d4b743fbabd47: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260924 09:56:16.528869 5530 log.cc:824] T 92bd23242c07444692da2ec596bbc066 P 48954e80951c451e8d16b2171fb479b0: Log is configured to *not* fsync() on all Append() calls
I20260924 09:56:16.530503 5532 tablet_bootstrap.cc:654] T 92bd23242c07444692da2ec596bbc066 P bd85c5a9f1c54df4b1efa5beef5b34b6: Neither blocks nor log segments found. Creating new log.
I20260924 09:56:16.531400 5532 log.cc:824] T 92bd23242c07444692da2ec596bbc066 P bd85c5a9f1c54df4b1efa5beef5b34b6: Log is configured to *not* fsync() on all Append() calls
I20260924 09:56:16.531749 5530 tablet_bootstrap.cc:492] T 92bd23242c07444692da2ec596bbc066 P 48954e80951c451e8d16b2171fb479b0: No bootstrap required, opened a new log
I20260924 09:56:16.532043 5530 ts_tablet_manager.cc:1397] T 92bd23242c07444692da2ec596bbc066 P 48954e80951c451e8d16b2171fb479b0: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.002s
I20260924 09:56:16.533803 5532 tablet_bootstrap.cc:492] T 92bd23242c07444692da2ec596bbc066 P bd85c5a9f1c54df4b1efa5beef5b34b6: No bootstrap required, opened a new log
I20260924 09:56:16.534008 5532 ts_tablet_manager.cc:1397] T 92bd23242c07444692da2ec596bbc066 P bd85c5a9f1c54df4b1efa5beef5b34b6: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260924 09:56:16.537490 5529 raft_consensus.cc:348] T 92bd23242c07444692da2ec596bbc066 P 89abbf11becb47b8a69d4b743fbabd47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48954e80951c451e8d16b2171fb479b0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44147 } } peers { permanent_uuid: "89abbf11becb47b8a69d4b743fbabd47" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39459 } } peers { permanent_uuid: "bd85c5a9f1c54df4b1efa5beef5b34b6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43047 } }
I20260924 09:56:16.537904 5529 raft_consensus.cc:374] T 92bd23242c07444692da2ec596bbc066 P 89abbf11becb47b8a69d4b743fbabd47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 09:56:16.538031 5529 raft_consensus.cc:729] T 92bd23242c07444692da2ec596bbc066 P 89abbf11becb47b8a69d4b743fbabd47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89abbf11becb47b8a69d4b743fbabd47, State: Initialized, Role: FOLLOWER
I20260924 09:56:16.538635 5529 consensus_queue.cc:260] T 92bd23242c07444692da2ec596bbc066 P 89abbf11becb47b8a69d4b743fbabd47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48954e80951c451e8d16b2171fb479b0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44147 } } peers { permanent_uuid: "89abbf11becb47b8a69d4b743fbabd47" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39459 } } peers { permanent_uuid: "bd85c5a9f1c54df4b1efa5beef5b34b6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43047 } }
I20260924 09:56:16.539722 5381 heartbeater.cc:503] Master 127.0.97.62:33771 was elected leader, sending a full tablet report...
I20260924 09:56:16.540248 5529 ts_tablet_manager.cc:1428] T 92bd23242c07444692da2ec596bbc066 P 89abbf11becb47b8a69d4b743fbabd47: Time spent starting tablet: real 0.012s user 0.007s sys 0.004s
I20260924 09:56:16.541718 5530 raft_consensus.cc:348] T 92bd23242c07444692da2ec596bbc066 P 48954e80951c451e8d16b2171fb479b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48954e80951c451e8d16b2171fb479b0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44147 } } peers { permanent_uuid: "89abbf11becb47b8a69d4b743fbabd47" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39459 } } peers { permanent_uuid: "bd85c5a9f1c54df4b1efa5beef5b34b6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43047 } }
I20260924 09:56:16.542104 5530 raft_consensus.cc:374] T 92bd23242c07444692da2ec596bbc066 P 48954e80951c451e8d16b2171fb479b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 09:56:16.542207 5530 raft_consensus.cc:729] T 92bd23242c07444692da2ec596bbc066 P 48954e80951c451e8d16b2171fb479b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48954e80951c451e8d16b2171fb479b0, State: Initialized, Role: FOLLOWER
I20260924 09:56:16.542270 5532 raft_consensus.cc:348] T 92bd23242c07444692da2ec596bbc066 P bd85c5a9f1c54df4b1efa5beef5b34b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48954e80951c451e8d16b2171fb479b0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44147 } } peers { permanent_uuid: "89abbf11becb47b8a69d4b743fbabd47" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39459 } } peers { permanent_uuid: "bd85c5a9f1c54df4b1efa5beef5b34b6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43047 } }
I20260924 09:56:16.542665 5532 raft_consensus.cc:374] T 92bd23242c07444692da2ec596bbc066 P bd85c5a9f1c54df4b1efa5beef5b34b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 09:56:16.542747 5532 raft_consensus.cc:729] T 92bd23242c07444692da2ec596bbc066 P bd85c5a9f1c54df4b1efa5beef5b34b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd85c5a9f1c54df4b1efa5beef5b34b6, State: Initialized, Role: FOLLOWER
I20260924 09:56:16.542763 5530 consensus_queue.cc:260] T 92bd23242c07444692da2ec596bbc066 P 48954e80951c451e8d16b2171fb479b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48954e80951c451e8d16b2171fb479b0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44147 } } peers { permanent_uuid: "89abbf11becb47b8a69d4b743fbabd47" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39459 } } peers { permanent_uuid: "bd85c5a9f1c54df4b1efa5beef5b34b6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43047 } }
I20260924 09:56:16.543267 5532 consensus_queue.cc:260] T 92bd23242c07444692da2ec596bbc066 P bd85c5a9f1c54df4b1efa5beef5b34b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48954e80951c451e8d16b2171fb479b0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44147 } } peers { permanent_uuid: "89abbf11becb47b8a69d4b743fbabd47" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39459 } } peers { permanent_uuid: "bd85c5a9f1c54df4b1efa5beef5b34b6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43047 } }
I20260924 09:56:16.544303 5251 heartbeater.cc:503] Master 127.0.97.62:33771 was elected leader, sending a full tablet report...
I20260924 09:56:16.545106 5532 ts_tablet_manager.cc:1428] T 92bd23242c07444692da2ec596bbc066 P bd85c5a9f1c54df4b1efa5beef5b34b6: Time spent starting tablet: real 0.011s user 0.006s sys 0.004s
I20260924 09:56:16.546060 5511 heartbeater.cc:503] Master 127.0.97.62:33771 was elected leader, sending a full tablet report...
I20260924 09:56:16.546690 5530 ts_tablet_manager.cc:1428] T 92bd23242c07444692da2ec596bbc066 P 48954e80951c451e8d16b2171fb479b0: Time spent starting tablet: real 0.015s user 0.004s sys 0.009s
I20260924 09:56:16.607214 5536 raft_consensus.cc:482] T 92bd23242c07444692da2ec596bbc066 P 48954e80951c451e8d16b2171fb479b0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 09:56:16.607486 5536 raft_consensus.cc:504] T 92bd23242c07444692da2ec596bbc066 P 48954e80951c451e8d16b2171fb479b0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48954e80951c451e8d16b2171fb479b0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44147 } } peers { permanent_uuid: "89abbf11becb47b8a69d4b743fbabd47" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39459 } } peers { permanent_uuid: "bd85c5a9f1c54df4b1efa5beef5b34b6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43047 } }
I20260924 09:56:16.608093 5535 raft_consensus.cc:482] T 92bd23242c07444692da2ec596bbc066 P 89abbf11becb47b8a69d4b743fbabd47 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 09:56:16.608356 5535 raft_consensus.cc:504] T 92bd23242c07444692da2ec596bbc066 P 89abbf11becb47b8a69d4b743fbabd47 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48954e80951c451e8d16b2171fb479b0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44147 } } peers { permanent_uuid: "89abbf11becb47b8a69d4b743fbabd47" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39459 } } peers { permanent_uuid: "bd85c5a9f1c54df4b1efa5beef5b34b6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43047 } }
I20260924 09:56:16.609355 5536 leader_election.cc:288] T 92bd23242c07444692da2ec596bbc066 P 48954e80951c451e8d16b2171fb479b0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 89abbf11becb47b8a69d4b743fbabd47 (127.0.97.2:39459), bd85c5a9f1c54df4b1efa5beef5b34b6 (127.0.97.1:43047)
I20260924 09:56:16.609871 5535 leader_election.cc:288] T 92bd23242c07444692da2ec596bbc066 P 89abbf11becb47b8a69d4b743fbabd47 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 48954e80951c451e8d16b2171fb479b0 (127.0.97.3:44147), bd85c5a9f1c54df4b1efa5beef5b34b6 (127.0.97.1:43047)
I20260924 09:56:16.612027 5513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:16.609629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:39459 (local address 127.0.97.3:50013)
0924 09:56:16.609805 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0924 09:56:16.609822 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 09:56:16.609926 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:56:16.610453 (+ 527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:56:16.610457 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:56:16.610482 (+ 25us) client_negotiation.cc:189] Negotiated authn=SASL
0924 09:56:16.611098 (+ 616us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 09:56:16.611102 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 09:56:16.611121 (+ 19us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 09:56:16.611170 (+ 49us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 09:56:16.611613 (+ 443us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 09:56:16.611618 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 09:56:16.611624 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 09:56:16.611713 (+ 89us) client_negotiation.cc:240] Negotiation successful
0924 09:56:16.611766 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64}
I20260924 09:56:16.613236 5526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:16.609616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:50013 (local address 127.0.97.2:39459)
0924 09:56:16.609797 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0924 09:56:16.609803 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 09:56:16.609892 (+ 89us) server_negotiation.cc:407] Connection header received
0924 09:56:16.610057 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:56:16.610062 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:56:16.610118 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:56:16.610228 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:56:16.611385 (+ 1157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:56:16.611391 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:56:16.611396 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 09:56:16.611415 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:56:16.611453 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 09:56:16.611467 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:56:16.611568 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0924 09:56:16.612913 (+ 1345us) server_negotiation.cc:299] Negotiation successful
0924 09:56:16.612971 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 09:56:16.615016 5513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:16.609963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43047 (local address 127.0.97.3:58691)
0924 09:56:16.612226 (+ 2263us) negotiation.cc:107] Waiting for socket to connect
0924 09:56:16.612243 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 09:56:16.612324 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:56:16.613561 (+ 1237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:56:16.613566 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:56:16.613590 (+ 24us) client_negotiation.cc:189] Negotiated authn=SASL
0924 09:56:16.614149 (+ 559us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 09:56:16.614153 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 09:56:16.614169 (+ 16us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 09:56:16.614222 (+ 53us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 09:56:16.614592 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 09:56:16.614596 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 09:56:16.614602 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 09:56:16.614683 (+ 81us) client_negotiation.cc:240] Negotiation successful
0924 09:56:16.614730 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2163,"threads_started":1}
I20260924 09:56:16.615644 5527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:16.610513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:58691 (local address 127.0.97.1:43047)
0924 09:56:16.610810 (+ 297us) server_negotiation.cc:206] Beginning negotiation
0924 09:56:16.610816 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 09:56:16.612412 (+ 1596us) server_negotiation.cc:407] Connection header received
0924 09:56:16.612474 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:56:16.612480 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:56:16.612538 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:56:16.612617 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:56:16.614378 (+ 1761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:56:16.614384 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:56:16.614389 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 09:56:16.614408 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:56:16.614442 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 09:56:16.614457 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:56:16.614551 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0924 09:56:16.615317 (+ 766us) server_negotiation.cc:299] Negotiation successful
0924 09:56:16.615376 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":185}
I20260924 09:56:16.618404 5525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:16.610628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:56625 (local address 127.0.97.3:44147)
0924 09:56:16.610925 (+ 297us) server_negotiation.cc:206] Beginning negotiation
0924 09:56:16.610930 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 09:56:16.616452 (+ 5522us) server_negotiation.cc:407] Connection header received
0924 09:56:16.616631 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:56:16.616636 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:56:16.616679 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:56:16.616738 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:56:16.617769 (+ 1031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:56:16.617775 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:56:16.617780 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 09:56:16.617797 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:56:16.617830 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 09:56:16.617844 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:56:16.617927 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0924 09:56:16.618112 (+ 185us) server_negotiation.cc:299] Negotiation successful
0924 09:56:16.618162 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":204}
I20260924 09:56:16.619215 5539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:16.610257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:44147 (local address 127.0.97.2:56625)
0924 09:56:16.616367 (+ 6110us) negotiation.cc:107] Waiting for socket to connect
0924 09:56:16.616384 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 09:56:16.616515 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:56:16.616913 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:56:16.616917 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:56:16.616937 (+ 20us) client_negotiation.cc:189] Negotiated authn=SASL
0924 09:56:16.617555 (+ 618us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 09:56:16.617559 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 09:56:16.617577 (+ 18us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 09:56:16.617622 (+ 45us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 09:56:16.618001 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 09:56:16.618006 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 09:56:16.618011 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 09:56:16.618757 (+ 746us) client_negotiation.cc:240] Negotiation successful
0924 09:56:16.618809 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5997,"thread_start_us":5869,"threads_started":1}
I20260924 09:56:16.620328 5466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92bd23242c07444692da2ec596bbc066" candidate_uuid: "89abbf11becb47b8a69d4b743fbabd47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48954e80951c451e8d16b2171fb479b0" is_pre_election: true
I20260924 09:56:16.620807 5466 raft_consensus.cc:2454] T 92bd23242c07444692da2ec596bbc066 P 48954e80951c451e8d16b2171fb479b0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89abbf11becb47b8a69d4b743fbabd47 in term 0.
I20260924 09:56:16.621663 5527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:16.611046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:35869 (local address 127.0.97.1:43047)
0924 09:56:16.615843 (+ 4797us) server_negotiation.cc:206] Beginning negotiation
0924 09:56:16.615848 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 09:56:16.619752 (+ 3904us) server_negotiation.cc:407] Connection header received
0924 09:56:16.619920 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:56:16.619926 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:56:16.619993 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:56:16.620054 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:56:16.620937 (+ 883us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:56:16.620943 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:56:16.620946 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 09:56:16.620962 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:56:16.620990 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 09:56:16.621004 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:56:16.621087 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0924 09:56:16.621294 (+ 207us) server_negotiation.cc:299] Negotiation successful
0924 09:56:16.621345 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4712,"threads_started":1}
I20260924 09:56:16.621598 5269 leader_election.cc:302] T 92bd23242c07444692da2ec596bbc066 P 89abbf11becb47b8a69d4b743fbabd47 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48954e80951c451e8d16b2171fb479b0, 89abbf11becb47b8a69d4b743fbabd47; no voters:
I20260924 09:56:16.622004 5535 raft_consensus.cc:2781] T 92bd23242c07444692da2ec596bbc066 P 89abbf11becb47b8a69d4b743fbabd47 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 09:56:16.622148 5535 raft_consensus.cc:482] T 92bd23242c07444692da2ec596bbc066 P 89abbf11becb47b8a69d4b743fbabd47 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 09:56:16.622249 5535 raft_consensus.cc:3037] T 92bd23242c07444692da2ec596bbc066 P 89abbf11becb47b8a69d4b743fbabd47 [term 0 FOLLOWER]: Advancing to term 1
I20260924 09:56:16.624614 5539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:16.610394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43047 (local address 127.0.97.2:35869)
0924 09:56:16.619672 (+ 9278us) negotiation.cc:107] Waiting for socket to connect
0924 09:56:16.619688 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 09:56:16.619815 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:56:16.620235 (+ 420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:56:16.620239 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:56:16.620259 (+ 20us) client_negotiation.cc:189] Negotiated authn=SASL
0924 09:56:16.620737 (+ 478us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 09:56:16.620740 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 09:56:16.620754 (+ 14us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 09:56:16.620792 (+ 38us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 09:56:16.621165 (+ 373us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 09:56:16.621170 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 09:56:16.621174 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 09:56:16.624330 (+ 3156us) client_negotiation.cc:240] Negotiation successful
0924 09:56:16.624377 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9168,"thread_start_us":13654,"threads_started":1}
I20260924 09:56:16.625571 5535 raft_consensus.cc:504] T 92bd23242c07444692da2ec596bbc066 P 89abbf11becb47b8a69d4b743fbabd47 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48954e80951c451e8d16b2171fb479b0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44147 } } peers { permanent_uuid: "89abbf11becb47b8a69d4b743fbabd47" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39459 } } peers { permanent_uuid: "bd85c5a9f1c54df4b1efa5beef5b34b6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43047 } }
I20260924 09:56:16.625839 5336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92bd23242c07444692da2ec596bbc066" candidate_uuid: "48954e80951c451e8d16b2171fb479b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89abbf11becb47b8a69d4b743fbabd47" is_pre_election: true
I20260924 09:56:16.626154 5535 leader_election.cc:288] T 92bd23242c07444692da2ec596bbc066 P 89abbf11becb47b8a69d4b743fbabd47 [CANDIDATE]: Term 1 election: Requested vote from peers 48954e80951c451e8d16b2171fb479b0 (127.0.97.3:44147), bd85c5a9f1c54df4b1efa5beef5b34b6 (127.0.97.1:43047)
I20260924 09:56:16.626288 5336 raft_consensus.cc:2379] T 92bd23242c07444692da2ec596bbc066 P 89abbf11becb47b8a69d4b743fbabd47 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 48954e80951c451e8d16b2171fb479b0 in current term 1: Already voted for candidate 89abbf11becb47b8a69d4b743fbabd47 in this term.
I20260924 09:56:16.626461 5466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92bd23242c07444692da2ec596bbc066" candidate_uuid: "89abbf11becb47b8a69d4b743fbabd47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48954e80951c451e8d16b2171fb479b0"
I20260924 09:56:16.626634 5466 raft_consensus.cc:3037] T 92bd23242c07444692da2ec596bbc066 P 48954e80951c451e8d16b2171fb479b0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 09:56:16.626587 5206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92bd23242c07444692da2ec596bbc066" candidate_uuid: "89abbf11becb47b8a69d4b743fbabd47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd85c5a9f1c54df4b1efa5beef5b34b6" is_pre_election: true
I20260924 09:56:16.627022 5206 raft_consensus.cc:2454] T 92bd23242c07444692da2ec596bbc066 P bd85c5a9f1c54df4b1efa5beef5b34b6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89abbf11becb47b8a69d4b743fbabd47 in term 0.
I20260924 09:56:16.627249 5205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92bd23242c07444692da2ec596bbc066" candidate_uuid: "48954e80951c451e8d16b2171fb479b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd85c5a9f1c54df4b1efa5beef5b34b6" is_pre_election: true
I20260924 09:56:16.627563 5205 raft_consensus.cc:2454] T 92bd23242c07444692da2ec596bbc066 P bd85c5a9f1c54df4b1efa5beef5b34b6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48954e80951c451e8d16b2171fb479b0 in term 0.
I20260924 09:56:16.628002 5402 leader_election.cc:302] T 92bd23242c07444692da2ec596bbc066 P 48954e80951c451e8d16b2171fb479b0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 48954e80951c451e8d16b2171fb479b0, bd85c5a9f1c54df4b1efa5beef5b34b6; no voters: 89abbf11becb47b8a69d4b743fbabd47
I20260924 09:56:16.628151 5205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92bd23242c07444692da2ec596bbc066" candidate_uuid: "89abbf11becb47b8a69d4b743fbabd47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd85c5a9f1c54df4b1efa5beef5b34b6"
I20260924 09:56:16.628328 5205 raft_consensus.cc:3037] T 92bd23242c07444692da2ec596bbc066 P bd85c5a9f1c54df4b1efa5beef5b34b6 [term 0 FOLLOWER]: Advancing to term 1
I20260924 09:56:16.629345 5466 raft_consensus.cc:2454] T 92bd23242c07444692da2ec596bbc066 P 48954e80951c451e8d16b2171fb479b0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89abbf11becb47b8a69d4b743fbabd47 in term 1.
I20260924 09:56:16.629500 5536 raft_consensus.cc:2741] T 92bd23242c07444692da2ec596bbc066 P 48954e80951c451e8d16b2171fb479b0 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20260924 09:56:16.629684 5269 leader_election.cc:302] T 92bd23242c07444692da2ec596bbc066 P 89abbf11becb47b8a69d4b743fbabd47 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48954e80951c451e8d16b2171fb479b0, 89abbf11becb47b8a69d4b743fbabd47; no voters:
I20260924 09:56:16.630331 5535 raft_consensus.cc:2781] T 92bd23242c07444692da2ec596bbc066 P 89abbf11becb47b8a69d4b743fbabd47 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 09:56:16.630707 5535 raft_consensus.cc:686] T 92bd23242c07444692da2ec596bbc066 P 89abbf11becb47b8a69d4b743fbabd47 [term 1 LEADER]: Becoming Leader. State: Replica: 89abbf11becb47b8a69d4b743fbabd47, State: Running, Role: LEADER
I20260924 09:56:16.630788 5205 raft_consensus.cc:2454] T 92bd23242c07444692da2ec596bbc066 P bd85c5a9f1c54df4b1efa5beef5b34b6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89abbf11becb47b8a69d4b743fbabd47 in term 1.
I20260924 09:56:16.631112 5535 consensus_queue.cc:237] T 92bd23242c07444692da2ec596bbc066 P 89abbf11becb47b8a69d4b743fbabd47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48954e80951c451e8d16b2171fb479b0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44147 } } peers { permanent_uuid: "89abbf11becb47b8a69d4b743fbabd47" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39459 } } peers { permanent_uuid: "bd85c5a9f1c54df4b1efa5beef5b34b6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43047 } }
W20260924 09:56:16.635142 5252 tablet.cc:2369] T 92bd23242c07444692da2ec596bbc066 P bd85c5a9f1c54df4b1efa5beef5b34b6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 09:56:16.635366 5064 catalog_manager.cc:5495] T 92bd23242c07444692da2ec596bbc066 P 89abbf11becb47b8a69d4b743fbabd47 reported cstate change: term changed from 0 to 1, leader changed from <none> to 89abbf11becb47b8a69d4b743fbabd47 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "89abbf11becb47b8a69d4b743fbabd47" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48954e80951c451e8d16b2171fb479b0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "89abbf11becb47b8a69d4b743fbabd47" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39459 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bd85c5a9f1c54df4b1efa5beef5b34b6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43047 } health_report { overall_health: UNKNOWN } } }
I20260924 09:56:16.653940 5526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:16.652336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47444 (local address 127.0.97.2:39459)
0924 09:56:16.652482 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0924 09:56:16.652487 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 09:56:16.652503 (+ 16us) server_negotiation.cc:407] Connection header received
0924 09:56:16.652639 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:56:16.652642 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:56:16.652726 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:56:16.652806 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:56:16.653483 (+ 677us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:56:16.653487 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:56:16.653490 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 09:56:16.653503 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:56:16.653530 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 09:56:16.653540 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:56:16.653641 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0924 09:56:16.653704 (+ 63us) server_negotiation.cc:299] Negotiation successful
0924 09:56:16.653744 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 09:56:16.661170 5466 raft_consensus.cc:1261] T 92bd23242c07444692da2ec596bbc066 P 48954e80951c451e8d16b2171fb479b0 [term 1 FOLLOWER]: Refusing update from remote peer 89abbf11becb47b8a69d4b743fbabd47: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 09:56:16.661191 5205 raft_consensus.cc:1261] T 92bd23242c07444692da2ec596bbc066 P bd85c5a9f1c54df4b1efa5beef5b34b6 [term 1 FOLLOWER]: Refusing update from remote peer 89abbf11becb47b8a69d4b743fbabd47: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 09:56:16.661962 5542 consensus_queue.cc:1035] T 92bd23242c07444692da2ec596bbc066 P 89abbf11becb47b8a69d4b743fbabd47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48954e80951c451e8d16b2171fb479b0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 09:56:16.662261 5535 consensus_queue.cc:1035] T 92bd23242c07444692da2ec596bbc066 P 89abbf11becb47b8a69d4b743fbabd47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd85c5a9f1c54df4b1efa5beef5b34b6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 09:56:16.687978 5065 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44112:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 09:56:16.688200 5065 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44112:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 09:56:16.691179 5065 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6e2ddf92c18c49e2bc730ec50830f8c6: Sending DeleteTablet for 3 replicas of tablet 92bd23242c07444692da2ec596bbc066
I20260924 09:56:16.692158 5446 tablet_service.cc:1514] Processing DeleteTablet for tablet 92bd23242c07444692da2ec596bbc066 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 09:56:16 UTC) from {username='slave'} at 127.0.0.1:34314
I20260924 09:56:16.692180 5316 tablet_service.cc:1514] Processing DeleteTablet for tablet 92bd23242c07444692da2ec596bbc066 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 09:56:16 UTC) from {username='slave'} at 127.0.0.1:47434
I20260924 09:56:16.693050 5556 tablet_replica.cc:330] T 92bd23242c07444692da2ec596bbc066 P 89abbf11becb47b8a69d4b743fbabd47: stopping tablet replica
I20260924 09:56:16.693264 5186 tablet_service.cc:1514] Processing DeleteTablet for tablet 92bd23242c07444692da2ec596bbc066 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 09:56:16 UTC) from {username='slave'} at 127.0.0.1:44120
I20260924 09:56:16.693500 5556 raft_consensus.cc:2229] T 92bd23242c07444692da2ec596bbc066 P 89abbf11becb47b8a69d4b743fbabd47 [term 1 LEADER]: Raft consensus shutting down.
I20260924 09:56:16.694092 5556 raft_consensus.cc:2258] T 92bd23242c07444692da2ec596bbc066 P 89abbf11becb47b8a69d4b743fbabd47 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 09:56:16.694759 5555 tablet_replica.cc:330] T 92bd23242c07444692da2ec596bbc066 P 48954e80951c451e8d16b2171fb479b0: stopping tablet replica
I20260924 09:56:16.695227 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu pid 5124
I20260924 09:56:16.695840 5557 tablet_replica.cc:330] T 92bd23242c07444692da2ec596bbc066 P bd85c5a9f1c54df4b1efa5beef5b34b6: stopping tablet replica
I20260924 09:56:16.696069 5556 ts_tablet_manager.cc:1905] T 92bd23242c07444692da2ec596bbc066 P 89abbf11becb47b8a69d4b743fbabd47: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 09:56:16.696192 5557 raft_consensus.cc:2229] T 92bd23242c07444692da2ec596bbc066 P bd85c5a9f1c54df4b1efa5beef5b34b6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 09:56:16.696641 5557 raft_consensus.cc:2258] T 92bd23242c07444692da2ec596bbc066 P bd85c5a9f1c54df4b1efa5beef5b34b6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 09:56:16.695120 5555 raft_consensus.cc:2229] T 92bd23242c07444692da2ec596bbc066 P 48954e80951c451e8d16b2171fb479b0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 09:56:16.697289 5555 raft_consensus.cc:2258] T 92bd23242c07444692da2ec596bbc066 P 48954e80951c451e8d16b2171fb479b0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 09:56:16.698597 5541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:16.696411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44132 (local address 127.0.97.1:43047)
0924 09:56:16.696588 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0924 09:56:16.696593 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 09:56:16.696619 (+ 26us) server_negotiation.cc:407] Connection header received
0924 09:56:16.696669 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:56:16.696673 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:56:16.696737 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:56:16.696824 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:56:16.697731 (+ 907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:56:16.697736 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:56:16.697741 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 09:56:16.697759 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:56:16.697803 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 09:56:16.697818 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:56:16.697934 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0924 09:56:16.698171 (+ 237us) server_negotiation.cc:299] Negotiation successful
0924 09:56:16.698226 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 09:56:16.699185 5557 ts_tablet_manager.cc:1905] T 92bd23242c07444692da2ec596bbc066 P bd85c5a9f1c54df4b1efa5beef5b34b6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 09:56:16.699529 5246 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 09:56:16.699601 5555 ts_tablet_manager.cc:1905] T 92bd23242c07444692da2ec596bbc066 P 48954e80951c451e8d16b2171fb479b0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 09:56:16.703227 5556 ts_tablet_manager.cc:1918] T 92bd23242c07444692da2ec596bbc066 P 89abbf11becb47b8a69d4b743fbabd47: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 09:56:16.703455 5556 log.cc:1196] T 92bd23242c07444692da2ec596bbc066 P 89abbf11becb47b8a69d4b743fbabd47: Deleting WAL directory at /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790243735634835-388-0/minicluster-data/ts-1/wal/wals/92bd23242c07444692da2ec596bbc066
I20260924 09:56:16.704007 5556 ts_tablet_manager.cc:1939] T 92bd23242c07444692da2ec596bbc066 P 89abbf11becb47b8a69d4b743fbabd47: Deleting consensus metadata
I20260924 09:56:16.705713 5052 catalog_manager.cc:4841] TS 89abbf11becb47b8a69d4b743fbabd47 (127.0.97.2:39459): tablet 92bd23242c07444692da2ec596bbc066 (table test-table [id=9ca7b23a01194200b2192105e8d1c22d]) successfully deleted
I20260924 09:56:16.706739 5555 ts_tablet_manager.cc:1918] T 92bd23242c07444692da2ec596bbc066 P 48954e80951c451e8d16b2171fb479b0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 09:56:16.707034 5555 log.cc:1196] T 92bd23242c07444692da2ec596bbc066 P 48954e80951c451e8d16b2171fb479b0: Deleting WAL directory at /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790243735634835-388-0/minicluster-data/ts-2/wal/wals/92bd23242c07444692da2ec596bbc066
I20260924 09:56:16.707722 5555 ts_tablet_manager.cc:1939] T 92bd23242c07444692da2ec596bbc066 P 48954e80951c451e8d16b2171fb479b0: Deleting consensus metadata
I20260924 09:56:16.709458 5049 catalog_manager.cc:4841] TS 48954e80951c451e8d16b2171fb479b0 (127.0.97.3:44147): tablet 92bd23242c07444692da2ec596bbc066 (table test-table [id=9ca7b23a01194200b2192105e8d1c22d]) successfully deleted
I20260924 09:56:16.808315 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu with pid 5124
W20260924 09:56:16.821851 5052 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:43047 (error 108)
I20260924 09:56:16.821983 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu pid 5254
I20260924 09:56:16.823735 5528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:16.823285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43047 (local address 127.0.0.1:44140)
0924 09:56:16.823442 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0924 09:56:16.823569 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:43047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260924 09:56:16.823975 5052 catalog_manager.cc:4584] TS bd85c5a9f1c54df4b1efa5beef5b34b6 (127.0.97.1:43047): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 92bd23242c07444692da2ec596bbc066: Network error: Client connection negotiation failed: client connection to 127.0.97.1:43047: connect: Connection refused (error 111)
I20260924 09:56:16.825109 5526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:16.823132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47450 (local address 127.0.97.2:39459)
0924 09:56:16.823317 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0924 09:56:16.823323 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 09:56:16.823342 (+ 19us) server_negotiation.cc:407] Connection header received
0924 09:56:16.823408 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:56:16.823413 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:56:16.823472 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:56:16.823575 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:56:16.824571 (+ 996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:56:16.824575 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:56:16.824578 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 09:56:16.824592 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:56:16.824623 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 09:56:16.824634 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:56:16.824725 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0924 09:56:16.824818 (+ 93us) server_negotiation.cc:299] Negotiation successful
0924 09:56:16.824861 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 09:56:16.825791 5376 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 09:56:16.877445 5528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:16.876918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43047 (local address 127.0.0.1:44156)
0924 09:56:16.877124 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0924 09:56:16.877254 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:43047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260924 09:56:16.878381 5528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:16.878038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43047 (local address 127.0.0.1:44170)
0924 09:56:16.878166 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0924 09:56:16.878260 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:43047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260924 09:56:16.930097 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu with pid 5254
I20260924 09:56:16.938885 5528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:16.938346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43047 (local address 127.0.0.1:44184)
0924 09:56:16.938576 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0924 09:56:16.938698 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:43047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260924 09:56:16.939908 5528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:16.939535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43047 (local address 127.0.0.1:44200)
0924 09:56:16.939687 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0924 09:56:16.939768 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:43047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260924 09:56:16.944484 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu pid 5384
I20260924 09:56:16.946751 5525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:16.945104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34324 (local address 127.0.97.3:44147)
0924 09:56:16.945283 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0924 09:56:16.945287 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 09:56:16.945303 (+ 16us) server_negotiation.cc:407] Connection header received
0924 09:56:16.945419 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:56:16.945422 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:56:16.945492 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:56:16.945563 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:56:16.946268 (+ 705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:56:16.946272 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:56:16.946275 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 09:56:16.946289 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:56:16.946317 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 09:56:16.946327 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:56:16.946423 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0924 09:56:16.946457 (+ 34us) server_negotiation.cc:299] Negotiation successful
0924 09:56:16.946496 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260924 09:56:16.947314 5506 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 09:56:17.020468 5528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:17.019950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43047 (local address 127.0.0.1:44206)
0924 09:56:17.020184 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0924 09:56:17.020299 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:43047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260924 09:56:17.021507 5528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:17.021074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43047 (local address 127.0.0.1:44216)
0924 09:56:17.021270 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0924 09:56:17.021352 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:43047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260924 09:56:17.033146 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu with pid 5384
I20260924 09:56:17.046314 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu pid 5034
I20260924 09:56:17.047376 5095 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 09:56:17.148471 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu with pid 5034
2026-09-24T09:56:17Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (2442 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260924 09:56:17.183813 388 mini_oidc.cc:152] Starting JWKS server
I20260924 09:56:17.186213 388 webserver.cc:460] Webserver started at https://127.0.0.1:45959/ using document root <none> and password file <none>
I20260924 09:56:17.186463 388 mini_oidc.cc:189] Starting OIDC Discovery server
I20260924 09:56:17.187036 388 webserver.cc:460] Webserver started at http://127.0.0.1:37649/ using document root <none> and password file <none>
2026-09-24T09:56:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T09:56:17Z Disabled control of system clock
I20260924 09:56:17.209613 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
/tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790243735634835-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790243735634835-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790243735634835-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790243735634835-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36309
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:40701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:36309
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45959/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790243735634835-388-0/testchainca.cert with env {}
W20260924 09:56:17.512130 5569 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 09:56:17.512415 5569 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 09:56:17.512506 5569 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 09:56:17.519657 5569 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 09:56:17.519760 5569 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 09:56:17.519804 5569 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 09:56:17.519855 5569 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 09:56:17.519899 5569 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 09:56:17.519956 5569 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 09:56:17.519994 5569 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:45959/jwks.json
I20260924 09:56:17.529639 5569 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790243735634835-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790243735634835-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:36309
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36309
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45959/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790243735634835-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790243735634835-388-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790243735634835-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 09:56:17.530829 5569 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 09:56:17.532058 5569 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 09:56:17.542829 5574 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:56:17.542865 5575 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:56:17.543030 5577 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 09:56:17.543545 5569 server_base.cc:1034] running on GCE node
I20260924 09:56:17.544319 5569 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 09:56:17.545647 5569 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 09:56:17.546852 5569 hybrid_clock.cc:648] HybridClock initialized: now 1790243777546805 us; error 38 us; skew 500 ppm
I20260924 09:56:17.547169 5569 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 09:56:17.549242 5569 webserver.cc:460] Webserver started at http://127.0.97.62:39373/ using document root <none> and password file <none>
I20260924 09:56:17.549870 5569 fs_manager.cc:359] Metadata directory not provided
I20260924 09:56:17.549966 5569 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 09:56:17.550262 5569 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 09:56:17.552996 5569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790243735634835-388-0/minicluster-data/master-0/data/instance:
uuid: "1ec82613e4744bb9be2d2ae04342c5b1"
format_stamp: "Formatted at 2026-09-24 09:56:17 on dist-test-slave-w8d2"
server_key: "fc89938fd7ef33f33cecf9b1415c6d59"
server_key_iv: "082c4ec8c316cf8aa822224c8c4e43a6"
server_key_version: "encryptionkey@0"
I20260924 09:56:17.553723 5569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790243735634835-388-0/minicluster-data/master-0/wal/instance:
uuid: "1ec82613e4744bb9be2d2ae04342c5b1"
format_stamp: "Formatted at 2026-09-24 09:56:17 on dist-test-slave-w8d2"
server_key: "fc89938fd7ef33f33cecf9b1415c6d59"
server_key_iv: "082c4ec8c316cf8aa822224c8c4e43a6"
server_key_version: "encryptionkey@0"
I20260924 09:56:17.558382 5569 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260924 09:56:17.561517 5583 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 09:56:17.562764 5569 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260924 09:56:17.562913 5569 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790243735634835-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790243735634835-388-0/minicluster-data/master-0/wal
uuid: "1ec82613e4744bb9be2d2ae04342c5b1"
format_stamp: "Formatted at 2026-09-24 09:56:17 on dist-test-slave-w8d2"
server_key: "fc89938fd7ef33f33cecf9b1415c6d59"
server_key_iv: "082c4ec8c316cf8aa822224c8c4e43a6"
server_key_version: "encryptionkey@0"
I20260924 09:56:17.563092 5569 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790243735634835-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790243735634835-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790243735634835-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 09:56:17.585975 5569 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 09:56:17.586774 5569 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 09:56:17.587023 5569 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 09:56:17.603029 5569 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:36309
I20260924 09:56:17.603179 5634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:36309 every 8 connection(s)
I20260924 09:56:17.604321 5569 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790243735634835-388-0/minicluster-data/master-0/data/info.pb
I20260924 09:56:17.605335 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu as pid 5569
I20260924 09:56:17.605517 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790243735634835-388-0/minicluster-data/master-0/wal/instance
I20260924 09:56:17.606267 388 external_mini_cluster.cc:1428] Setting key d6a3b9a5fdc519d916c6d39b6b764773
I20260924 09:56:17.608816 5635 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 09:56:17.618728 5638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:17.608092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36238 (local address 127.0.97.62:36309)
0924 09:56:17.608982 (+ 890us) server_negotiation.cc:206] Beginning negotiation
0924 09:56:17.608993 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 09:56:17.609033 (+ 40us) server_negotiation.cc:407] Connection header received
0924 09:56:17.610588 (+ 1555us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:56:17.610608 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:56:17.611044 (+ 436us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:56:17.611433 (+ 389us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:56:17.611957 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:17.613234 (+ 1277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:17.614574 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:17.615184 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:17.615415 (+ 231us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:17.616579 (+ 1164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:56:17.616589 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:56:17.616602 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 09:56:17.617151 (+ 549us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:56:17.617205 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 09:56:17.617224 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:56:17.617338 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0924 09:56:17.617520 (+ 182us) server_negotiation.cc:299] Negotiation successful
0924 09:56:17.617787 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":459,"thread_start_us":269,"threads_started":1}
I20260924 09:56:17.619081 5635 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ec82613e4744bb9be2d2ae04342c5b1: Bootstrap starting.
I20260924 09:56:17.622321 5635 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1ec82613e4744bb9be2d2ae04342c5b1: Neither blocks nor log segments found. Creating new log.
I20260924 09:56:17.622349 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 09:56:17.623214 5635 log.cc:824] T 00000000000000000000000000000000 P 1ec82613e4744bb9be2d2ae04342c5b1: Log is configured to *not* fsync() on all Append() calls
I20260924 09:56:17.625905 5635 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ec82613e4744bb9be2d2ae04342c5b1: No bootstrap required, opened a new log
I20260924 09:56:17.632751 5635 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1ec82613e4744bb9be2d2ae04342c5b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ec82613e4744bb9be2d2ae04342c5b1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36309 } }
I20260924 09:56:17.633064 5635 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1ec82613e4744bb9be2d2ae04342c5b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 09:56:17.633160 5635 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1ec82613e4744bb9be2d2ae04342c5b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ec82613e4744bb9be2d2ae04342c5b1, State: Initialized, Role: FOLLOWER
I20260924 09:56:17.633673 5635 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1ec82613e4744bb9be2d2ae04342c5b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ec82613e4744bb9be2d2ae04342c5b1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36309 } }
I20260924 09:56:17.633869 5635 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1ec82613e4744bb9be2d2ae04342c5b1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 09:56:17.633986 5635 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1ec82613e4744bb9be2d2ae04342c5b1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 09:56:17.634124 5635 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1ec82613e4744bb9be2d2ae04342c5b1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 09:56:17.636214 5635 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1ec82613e4744bb9be2d2ae04342c5b1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ec82613e4744bb9be2d2ae04342c5b1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36309 } }
I20260924 09:56:17.636677 5635 leader_election.cc:302] T 00000000000000000000000000000000 P 1ec82613e4744bb9be2d2ae04342c5b1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1ec82613e4744bb9be2d2ae04342c5b1; no voters:
I20260924 09:56:17.637166 5635 leader_election.cc:288] T 00000000000000000000000000000000 P 1ec82613e4744bb9be2d2ae04342c5b1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 09:56:17.637300 5640 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1ec82613e4744bb9be2d2ae04342c5b1 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 09:56:17.638334 5640 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1ec82613e4744bb9be2d2ae04342c5b1 [term 1 LEADER]: Becoming Leader. State: Replica: 1ec82613e4744bb9be2d2ae04342c5b1, State: Running, Role: LEADER
I20260924 09:56:17.638893 5640 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1ec82613e4744bb9be2d2ae04342c5b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ec82613e4744bb9be2d2ae04342c5b1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36309 } }
I20260924 09:56:17.639132 5635 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1ec82613e4744bb9be2d2ae04342c5b1 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 09:56:17.641299 5642 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ec82613e4744bb9be2d2ae04342c5b1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1ec82613e4744bb9be2d2ae04342c5b1. Latest consensus state: current_term: 1 leader_uuid: "1ec82613e4744bb9be2d2ae04342c5b1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ec82613e4744bb9be2d2ae04342c5b1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36309 } } }
I20260924 09:56:17.641633 5642 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ec82613e4744bb9be2d2ae04342c5b1 [sys.catalog]: This master's current role is: LEADER
I20260924 09:56:17.642112 5641 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ec82613e4744bb9be2d2ae04342c5b1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1ec82613e4744bb9be2d2ae04342c5b1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ec82613e4744bb9be2d2ae04342c5b1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36309 } } }
I20260924 09:56:17.642539 5641 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ec82613e4744bb9be2d2ae04342c5b1 [sys.catalog]: This master's current role is: LEADER
I20260924 09:56:17.648578 5648 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 09:56:17.656464 5648 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 09:56:17.663113 5658 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260924 09:56:17.665937 5648 catalog_manager.cc:1347] Generated new cluster ID: c773588deb0e4018b121b1cf77254fa8
I20260924 09:56:17.666044 5648 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 09:56:17.685793 5648 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 09:56:17.687680 5648 catalog_manager.cc:1504] Loading token signing keys...
I20260924 09:56:17.701084 5648 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1ec82613e4744bb9be2d2ae04342c5b1: Generated new TSK 0
I20260924 09:56:17.701977 5648 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260924 09:56:17.736488 5666 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260924 09:56:17.736631 5666 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260924 09:56:17.737811 5638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:17.716568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36250 (local address 127.0.97.62:36309)
0924 09:56:17.716747 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0924 09:56:17.716753 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 09:56:17.717159 (+ 406us) server_negotiation.cc:407] Connection header received
0924 09:56:17.717438 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:56:17.717444 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:56:17.717528 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:56:17.717682 (+ 154us) server_negotiation.cc:226] Negotiated authn=JWT
0924 09:56:17.718132 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:17.719279 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:17.720653 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:17.721410 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:17.721533 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:17.722030 (+ 497us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 09:56:17.737116 (+ 15086us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45959/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0924 09:56:17.737487 (+ 371us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:36250: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45959/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":63}
W20260924 09:56:17.738003 5638 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:36250: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45959/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260924 09:56:17.739450 5665 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 09:56:17.716446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36309 (local address 127.0.0.1:36250)
0924 09:56:17.717052 (+ 606us) negotiation.cc:107] Waiting for socket to connect
0924 09:56:17.717077 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0924 09:56:17.717228 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:56:17.717690 (+ 462us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:56:17.717694 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:56:17.717703 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0924 09:56:17.718002 (+ 299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:17.718009 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:17.719402 (+ 1393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:17.719406 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:17.720525 (+ 1119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:17.720536 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:17.721539 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:17.721592 (+ 53us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:17.721736 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:17.721791 (+ 55us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 09:56:17.737429 (+ 15638us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45959/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0924 09:56:17.737775 (+ 346us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.97.62:36309: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45959/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":310,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.0.97.62:36309: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45959/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260924 09:56:17.741267 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu pid 5569
I20260924 09:56:17.742367 5630 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 09:56:17.832147 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu with pid 5569
2026-09-24T09:56:17Z chronyd exiting
I20260924 09:56:17.878750 388 test_util.cc:175] -----------------------------------------------
I20260924 09:56:17.878870 388 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790243735634835-388-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (699 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260924 09:56:17.882412 388 mini_oidc.cc:152] Starting JWKS server
I20260924 09:56:17.884160 388 webserver.cc:460] Webserver started at https://127.0.0.1:46189/ using document root <none> and password file <none>
I20260924 09:56:17.884379 388 mini_oidc.cc:189] Starting OIDC Discovery server
I20260924 09:56:17.884919 388 webserver.cc:460] Webserver started at http://127.0.0.1:34609/ using document root <none> and password file <none>
2026-09-24T09:56:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T09:56:17Z Disabled control of system clock
I20260924 09:56:17.906939 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
/tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790243735634835-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790243735634835-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790243735634835-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790243735634835-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35591
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:39653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:35591
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46189/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790243735634835-388-0/testchainca.cert with env {}
W20260924 09:56:18.207919 5673 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 09:56:18.208195 5673 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 09:56:18.208279 5673 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 09:56:18.215087 5673 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 09:56:18.215207 5673 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 09:56:18.215281 5673 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 09:56:18.215338 5673 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 09:56:18.215380 5673 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 09:56:18.215440 5673 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 09:56:18.215476 5673 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:46189/jwks.json
I20260924 09:56:18.224735 5673 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790243735634835-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790243735634835-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:35591
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35591
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46189/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790243735634835-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790243735634835-388-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790243735634835-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 09:56:18.225878 5673 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 09:56:18.227020 5673 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 09:56:18.237820 5678 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:56:18.237779 5679 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:56:18.237849 5681 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 09:56:18.238713 5673 server_base.cc:1034] running on GCE node
I20260924 09:56:18.239485 5673 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 09:56:18.240808 5673 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 09:56:18.241997 5673 hybrid_clock.cc:648] HybridClock initialized: now 1790243778241950 us; error 37 us; skew 500 ppm
I20260924 09:56:18.242313 5673 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 09:56:18.244393 5673 webserver.cc:460] Webserver started at http://127.0.97.62:43389/ using document root <none> and password file <none>
I20260924 09:56:18.245005 5673 fs_manager.cc:359] Metadata directory not provided
I20260924 09:56:18.245090 5673 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 09:56:18.245389 5673 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 09:56:18.248054 5673 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790243735634835-388-0/minicluster-data/master-0/data/instance:
uuid: "4d22dd96391a4bec99c10c771efd0330"
format_stamp: "Formatted at 2026-09-24 09:56:18 on dist-test-slave-w8d2"
server_key: "5bda7b184f38b2f4343f77072ae9ba4c"
server_key_iv: "1a4e32e3a03bde40f8a1b0581ea2935e"
server_key_version: "encryptionkey@0"
I20260924 09:56:18.248772 5673 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790243735634835-388-0/minicluster-data/master-0/wal/instance:
uuid: "4d22dd96391a4bec99c10c771efd0330"
format_stamp: "Formatted at 2026-09-24 09:56:18 on dist-test-slave-w8d2"
server_key: "5bda7b184f38b2f4343f77072ae9ba4c"
server_key_iv: "1a4e32e3a03bde40f8a1b0581ea2935e"
server_key_version: "encryptionkey@0"
I20260924 09:56:18.253340 5673 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 09:56:18.256379 5687 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 09:56:18.257588 5673 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 09:56:18.257733 5673 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790243735634835-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790243735634835-388-0/minicluster-data/master-0/wal
uuid: "4d22dd96391a4bec99c10c771efd0330"
format_stamp: "Formatted at 2026-09-24 09:56:18 on dist-test-slave-w8d2"
server_key: "5bda7b184f38b2f4343f77072ae9ba4c"
server_key_iv: "1a4e32e3a03bde40f8a1b0581ea2935e"
server_key_version: "encryptionkey@0"
I20260924 09:56:18.257915 5673 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790243735634835-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790243735634835-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790243735634835-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 09:56:18.274231 5673 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 09:56:18.275002 5673 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 09:56:18.275281 5673 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 09:56:18.291482 5673 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:35591
I20260924 09:56:18.291491 5738 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:35591 every 8 connection(s)
I20260924 09:56:18.292774 5673 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790243735634835-388-0/minicluster-data/master-0/data/info.pb
I20260924 09:56:18.296305 5739 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 09:56:18.301047 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu as pid 5673
I20260924 09:56:18.301263 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790243735634835-388-0/minicluster-data/master-0/wal/instance
I20260924 09:56:18.301961 388 external_mini_cluster.cc:1428] Setting key 71f05132651298de1e155d2d00c39066
I20260924 09:56:18.307044 5739 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d22dd96391a4bec99c10c771efd0330: Bootstrap starting.
I20260924 09:56:18.310303 5739 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4d22dd96391a4bec99c10c771efd0330: Neither blocks nor log segments found. Creating new log.
I20260924 09:56:18.311440 5739 log.cc:824] T 00000000000000000000000000000000 P 4d22dd96391a4bec99c10c771efd0330: Log is configured to *not* fsync() on all Append() calls
I20260924 09:56:18.314435 5739 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d22dd96391a4bec99c10c771efd0330: No bootstrap required, opened a new log
I20260924 09:56:18.315593 5742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:18.303788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49776 (local address 127.0.97.62:35591)
0924 09:56:18.304760 (+ 972us) server_negotiation.cc:206] Beginning negotiation
0924 09:56:18.304770 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 09:56:18.304814 (+ 44us) server_negotiation.cc:407] Connection header received
0924 09:56:18.306655 (+ 1841us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:56:18.306683 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:56:18.307183 (+ 500us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:56:18.307557 (+ 374us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:56:18.308053 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:18.309313 (+ 1260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:18.310942 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:18.311684 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:18.311812 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:18.312877 (+ 1065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:56:18.312886 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:56:18.312904 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 09:56:18.313565 (+ 661us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:56:18.313662 (+ 97us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 09:56:18.313686 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:56:18.313802 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0924 09:56:18.314013 (+ 211us) server_negotiation.cc:299] Negotiation successful
0924 09:56:18.314313 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":559,"thread_start_us":328,"threads_started":1}
I20260924 09:56:18.319350 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 09:56:18.323725 5739 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4d22dd96391a4bec99c10c771efd0330 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d22dd96391a4bec99c10c771efd0330" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35591 } }
I20260924 09:56:18.324047 5739 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4d22dd96391a4bec99c10c771efd0330 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 09:56:18.324136 5739 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4d22dd96391a4bec99c10c771efd0330 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d22dd96391a4bec99c10c771efd0330, State: Initialized, Role: FOLLOWER
I20260924 09:56:18.324621 5739 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4d22dd96391a4bec99c10c771efd0330 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d22dd96391a4bec99c10c771efd0330" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35591 } }
I20260924 09:56:18.324815 5739 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4d22dd96391a4bec99c10c771efd0330 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 09:56:18.324934 5739 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4d22dd96391a4bec99c10c771efd0330 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 09:56:18.325071 5739 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4d22dd96391a4bec99c10c771efd0330 [term 0 FOLLOWER]: Advancing to term 1
I20260924 09:56:18.327286 5739 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4d22dd96391a4bec99c10c771efd0330 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d22dd96391a4bec99c10c771efd0330" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35591 } }
I20260924 09:56:18.327755 5739 leader_election.cc:302] T 00000000000000000000000000000000 P 4d22dd96391a4bec99c10c771efd0330 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4d22dd96391a4bec99c10c771efd0330; no voters:
I20260924 09:56:18.328318 5739 leader_election.cc:288] T 00000000000000000000000000000000 P 4d22dd96391a4bec99c10c771efd0330 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 09:56:18.328435 5744 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4d22dd96391a4bec99c10c771efd0330 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 09:56:18.329068 5744 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4d22dd96391a4bec99c10c771efd0330 [term 1 LEADER]: Becoming Leader. State: Replica: 4d22dd96391a4bec99c10c771efd0330, State: Running, Role: LEADER
I20260924 09:56:18.329532 5744 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4d22dd96391a4bec99c10c771efd0330 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d22dd96391a4bec99c10c771efd0330" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35591 } }
I20260924 09:56:18.330171 5739 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4d22dd96391a4bec99c10c771efd0330 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 09:56:18.332177 5745 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d22dd96391a4bec99c10c771efd0330 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4d22dd96391a4bec99c10c771efd0330" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d22dd96391a4bec99c10c771efd0330" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35591 } } }
I20260924 09:56:18.332609 5745 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d22dd96391a4bec99c10c771efd0330 [sys.catalog]: This master's current role is: LEADER
I20260924 09:56:18.333215 5746 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d22dd96391a4bec99c10c771efd0330 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4d22dd96391a4bec99c10c771efd0330. Latest consensus state: current_term: 1 leader_uuid: "4d22dd96391a4bec99c10c771efd0330" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d22dd96391a4bec99c10c771efd0330" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35591 } } }
I20260924 09:56:18.333494 5746 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d22dd96391a4bec99c10c771efd0330 [sys.catalog]: This master's current role is: LEADER
I20260924 09:56:18.334534 5753 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 09:56:18.339618 5753 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 09:56:18.348179 5753 catalog_manager.cc:1347] Generated new cluster ID: f34ffc57829944169923bd8e121b1a25
I20260924 09:56:18.348295 5753 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 09:56:18.375186 5753 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 09:56:18.377014 5753 catalog_manager.cc:1504] Loading token signing keys...
I20260924 09:56:18.387780 5753 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4d22dd96391a4bec99c10c771efd0330: Generated new TSK 0
I20260924 09:56:18.388775 5753 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260924 09:56:18.436270 5769 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260924 09:56:18.436394 5769 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260924 09:56:18.437376 5742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:18.410621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49786 (local address 127.0.97.62:35591)
0924 09:56:18.410813 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0924 09:56:18.410819 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 09:56:18.411113 (+ 294us) server_negotiation.cc:407] Connection header received
0924 09:56:18.411335 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:56:18.411341 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:56:18.411437 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:56:18.411607 (+ 170us) server_negotiation.cc:226] Negotiated authn=JWT
0924 09:56:18.412012 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:18.413129 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:18.414393 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:18.415082 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:18.415208 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:18.415712 (+ 504us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 09:56:18.436731 (+ 21019us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46189/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0924 09:56:18.437091 (+ 360us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:49786: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46189/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":64}
W20260924 09:56:18.437522 5742 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:49786: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46189/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260924 09:56:18.437582 5768 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 09:56:18.410455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35591 (local address 127.0.0.1:49786)
0924 09:56:18.411018 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0924 09:56:18.411033 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 09:56:18.411145 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:56:18.411612 (+ 467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:56:18.411616 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:56:18.411626 (+ 10us) client_negotiation.cc:189] Negotiated authn=JWT
0924 09:56:18.411866 (+ 240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:18.411874 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:18.413244 (+ 1370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:18.413248 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:18.414248 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:18.414258 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:18.415184 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:18.415219 (+ 35us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:18.415422 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:18.415457 (+ 35us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 09:56:18.436962 (+ 21505us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46189/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0924 09:56:18.437210 (+ 248us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.97.62:35591: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46189/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":306,"threads_started":1}
I20260924 09:56:18.438589 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu pid 5673
I20260924 09:56:18.440011 5734 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 09:56:18.531057 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu with pid 5673
2026-09-24T09:56:18Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (700 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260924 09:56:18.582563 388 mini_oidc.cc:152] Starting JWKS server
I20260924 09:56:18.584349 388 webserver.cc:460] Webserver started at https://127.0.0.1:39255/ using document root <none> and password file <none>
I20260924 09:56:18.584535 388 mini_oidc.cc:189] Starting OIDC Discovery server
I20260924 09:56:18.585109 388 webserver.cc:460] Webserver started at http://127.0.0.1:46845/ using document root <none> and password file <none>
2026-09-24T09:56:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T09:56:18Z Disabled control of system clock
I20260924 09:56:18.607436 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
/tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790243735634835-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790243735634835-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790243735634835-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790243735634835-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42405
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:46547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:42405
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39255/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260924 09:56:18.904181 5776 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 09:56:18.904469 5776 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 09:56:18.904589 5776 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 09:56:18.911649 5776 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 09:56:18.911757 5776 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 09:56:18.911816 5776 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 09:56:18.911895 5776 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 09:56:18.911949 5776 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 09:56:18.912031 5776 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 09:56:18.912081 5776 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:39255/jwks.json
I20260924 09:56:18.921119 5776 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790243735634835-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790243735634835-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:42405
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42405
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39255/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790243735634835-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790243735634835-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 09:56:18.922328 5776 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 09:56:18.924129 5776 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 09:56:18.934638 5781 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:56:18.934710 5782 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:56:18.934681 5784 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 09:56:18.935583 5776 server_base.cc:1034] running on GCE node
I20260924 09:56:18.936342 5776 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 09:56:18.937668 5776 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 09:56:18.938870 5776 hybrid_clock.cc:648] HybridClock initialized: now 1790243778938818 us; error 44 us; skew 500 ppm
I20260924 09:56:18.939199 5776 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 09:56:18.941346 5776 webserver.cc:460] Webserver started at http://127.0.97.62:36313/ using document root <none> and password file <none>
I20260924 09:56:18.941996 5776 fs_manager.cc:359] Metadata directory not provided
I20260924 09:56:18.942092 5776 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 09:56:18.942404 5776 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 09:56:18.945267 5776 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790243735634835-388-0/minicluster-data/master-0/data/instance:
uuid: "2a562d982bdc43f083226fab9ce0f004"
format_stamp: "Formatted at 2026-09-24 09:56:18 on dist-test-slave-w8d2"
server_key: "f08e54cd25b41f53df43828f3c27d301"
server_key_iv: "86e3dffb59620cd703426ea38885b393"
server_key_version: "encryptionkey@0"
I20260924 09:56:18.946008 5776 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790243735634835-388-0/minicluster-data/master-0/wal/instance:
uuid: "2a562d982bdc43f083226fab9ce0f004"
format_stamp: "Formatted at 2026-09-24 09:56:18 on dist-test-slave-w8d2"
server_key: "f08e54cd25b41f53df43828f3c27d301"
server_key_iv: "86e3dffb59620cd703426ea38885b393"
server_key_version: "encryptionkey@0"
I20260924 09:56:18.950881 5776 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260924 09:56:18.954113 5790 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 09:56:18.955427 5776 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260924 09:56:18.955571 5776 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790243735634835-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790243735634835-388-0/minicluster-data/master-0/wal
uuid: "2a562d982bdc43f083226fab9ce0f004"
format_stamp: "Formatted at 2026-09-24 09:56:18 on dist-test-slave-w8d2"
server_key: "f08e54cd25b41f53df43828f3c27d301"
server_key_iv: "86e3dffb59620cd703426ea38885b393"
server_key_version: "encryptionkey@0"
I20260924 09:56:18.955754 5776 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790243735634835-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790243735634835-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790243735634835-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 09:56:18.977029 5776 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 09:56:18.977806 5776 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 09:56:18.978058 5776 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 09:56:18.994251 5776 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:42405
I20260924 09:56:18.994294 5841 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:42405 every 8 connection(s)
I20260924 09:56:18.995527 5776 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790243735634835-388-0/minicluster-data/master-0/data/info.pb
I20260924 09:56:18.998900 5842 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 09:56:19.001883 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu as pid 5776
I20260924 09:56:19.002043 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790243735634835-388-0/minicluster-data/master-0/wal/instance
I20260924 09:56:19.002625 388 external_mini_cluster.cc:1428] Setting key daa47ee70f9e3579f569a8a5160df92b
I20260924 09:56:19.010290 5842 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a562d982bdc43f083226fab9ce0f004: Bootstrap starting.
I20260924 09:56:19.013614 5842 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2a562d982bdc43f083226fab9ce0f004: Neither blocks nor log segments found. Creating new log.
I20260924 09:56:19.014632 5842 log.cc:824] T 00000000000000000000000000000000 P 2a562d982bdc43f083226fab9ce0f004: Log is configured to *not* fsync() on all Append() calls
I20260924 09:56:19.014938 5845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:19.004316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48208 (local address 127.0.97.62:42405)
0924 09:56:19.005211 (+ 895us) server_negotiation.cc:206] Beginning negotiation
0924 09:56:19.005221 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 09:56:19.005271 (+ 50us) server_negotiation.cc:407] Connection header received
0924 09:56:19.006739 (+ 1468us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:56:19.006760 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:56:19.007217 (+ 457us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:56:19.007581 (+ 364us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:56:19.008087 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:19.009304 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:19.010945 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:19.011605 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:19.011735 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:19.012818 (+ 1083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:56:19.012825 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:56:19.012837 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 09:56:19.013328 (+ 491us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:56:19.013408 (+ 80us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 09:56:19.013427 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:56:19.013506 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0924 09:56:19.013723 (+ 217us) server_negotiation.cc:299] Negotiation successful
0924 09:56:19.014044 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":526,"thread_start_us":321,"threads_started":1}
I20260924 09:56:19.017184 5842 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a562d982bdc43f083226fab9ce0f004: No bootstrap required, opened a new log
I20260924 09:56:19.018419 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 09:56:19.024917 5842 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2a562d982bdc43f083226fab9ce0f004 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a562d982bdc43f083226fab9ce0f004" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42405 } }
I20260924 09:56:19.025220 5842 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2a562d982bdc43f083226fab9ce0f004 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 09:56:19.025308 5842 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2a562d982bdc43f083226fab9ce0f004 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a562d982bdc43f083226fab9ce0f004, State: Initialized, Role: FOLLOWER
I20260924 09:56:19.025763 5842 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2a562d982bdc43f083226fab9ce0f004 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a562d982bdc43f083226fab9ce0f004" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42405 } }
I20260924 09:56:19.025985 5842 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2a562d982bdc43f083226fab9ce0f004 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 09:56:19.026108 5842 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2a562d982bdc43f083226fab9ce0f004 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 09:56:19.026259 5842 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2a562d982bdc43f083226fab9ce0f004 [term 0 FOLLOWER]: Advancing to term 1
I20260924 09:56:19.028313 5842 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2a562d982bdc43f083226fab9ce0f004 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a562d982bdc43f083226fab9ce0f004" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42405 } }
I20260924 09:56:19.028729 5842 leader_election.cc:302] T 00000000000000000000000000000000 P 2a562d982bdc43f083226fab9ce0f004 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2a562d982bdc43f083226fab9ce0f004; no voters:
I20260924 09:56:19.029234 5842 leader_election.cc:288] T 00000000000000000000000000000000 P 2a562d982bdc43f083226fab9ce0f004 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 09:56:19.029376 5847 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2a562d982bdc43f083226fab9ce0f004 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 09:56:19.029978 5847 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2a562d982bdc43f083226fab9ce0f004 [term 1 LEADER]: Becoming Leader. State: Replica: 2a562d982bdc43f083226fab9ce0f004, State: Running, Role: LEADER
I20260924 09:56:19.030464 5847 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2a562d982bdc43f083226fab9ce0f004 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a562d982bdc43f083226fab9ce0f004" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42405 } }
I20260924 09:56:19.031164 5842 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2a562d982bdc43f083226fab9ce0f004 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 09:56:19.033325 5849 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a562d982bdc43f083226fab9ce0f004 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2a562d982bdc43f083226fab9ce0f004. Latest consensus state: current_term: 1 leader_uuid: "2a562d982bdc43f083226fab9ce0f004" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a562d982bdc43f083226fab9ce0f004" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42405 } } }
I20260924 09:56:19.033759 5849 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a562d982bdc43f083226fab9ce0f004 [sys.catalog]: This master's current role is: LEADER
I20260924 09:56:19.033382 5848 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a562d982bdc43f083226fab9ce0f004 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2a562d982bdc43f083226fab9ce0f004" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a562d982bdc43f083226fab9ce0f004" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42405 } } }
I20260924 09:56:19.035638 5848 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a562d982bdc43f083226fab9ce0f004 [sys.catalog]: This master's current role is: LEADER
I20260924 09:56:19.038682 5854 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 09:56:19.043565 5854 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 09:56:19.051170 5854 catalog_manager.cc:1347] Generated new cluster ID: 939707852b9744d1af6dd019fe0c30d9
I20260924 09:56:19.051311 5854 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 09:56:19.081024 5854 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 09:56:19.082801 5854 catalog_manager.cc:1504] Loading token signing keys...
I20260924 09:56:19.092598 5854 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2a562d982bdc43f083226fab9ce0f004: Generated new TSK 0
I20260924 09:56:19.093547 5854 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 09:56:19.121788 5845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:19.113634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48218 (local address 127.0.97.62:42405)
0924 09:56:19.113816 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0924 09:56:19.113822 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 09:56:19.114211 (+ 389us) server_negotiation.cc:407] Connection header received
0924 09:56:19.114423 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:56:19.114428 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:56:19.114523 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:56:19.114689 (+ 166us) server_negotiation.cc:226] Negotiated authn=JWT
0924 09:56:19.115163 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:19.116273 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:19.117638 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:19.118308 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:19.118429 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:19.118886 (+ 457us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 09:56:19.121158 (+ 2272us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39255/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0924 09:56:19.121485 (+ 327us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:48218: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39255/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":55}
W20260924 09:56:19.121908 5845 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:48218: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39255/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260924 09:56:19.123656 5871 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 09:56:19.113483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42405 (local address 127.0.0.1:48218)
0924 09:56:19.114121 (+ 638us) negotiation.cc:107] Waiting for socket to connect
0924 09:56:19.114136 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 09:56:19.114252 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:56:19.114733 (+ 481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:56:19.114737 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:56:19.114747 (+ 10us) client_negotiation.cc:189] Negotiated authn=JWT
0924 09:56:19.115018 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:19.115026 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:19.116425 (+ 1399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:19.116430 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:19.117512 (+ 1082us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:19.117523 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:19.118421 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:19.118481 (+ 60us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:19.118596 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:19.118631 (+ 35us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 09:56:19.121361 (+ 2730us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39255/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0924 09:56:19.123206 (+ 1845us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.97.62:42405: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39255/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":517,"mutex_wait_us":1559,"thread_start_us":361,"threads_started":1}
I20260924 09:56:19.124812 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu pid 5776
I20260924 09:56:19.125969 5837 generic_service.cc:196] Checking for leaks (request via RPC)
E20260924 09:56:19.133201 5872 webserver.cc:533] Webserver: sslize failed:
E20260924 09:56:19.133292 5872 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260924 09:56:19.216265 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu with pid 5776
2026-09-24T09:56:19Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (684 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-24T09:56:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T09:56:19Z Disabled control of system clock
I20260924 09:56:19.289920 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
/tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790243735634835-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790243735634835-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790243735634835-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790243735634835-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34373
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:42973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:34373
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790243735634835-388-0/testchainca.cert with env {}
W20260924 09:56:19.591558 5877 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 09:56:19.591841 5877 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 09:56:19.591921 5877 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 09:56:19.598845 5877 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 09:56:19.598938 5877 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 09:56:19.598981 5877 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 09:56:19.599036 5877 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 09:56:19.599078 5877 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 09:56:19.599135 5877 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 09:56:19.599172 5877 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260924 09:56:19.608659 5877 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790243735634835-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790243735634835-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:34373
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34373
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790243735634835-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790243735634835-388-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790243735634835-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 09:56:19.609823 5877 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 09:56:19.610984 5877 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 09:56:19.621723 5882 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:56:19.621757 5883 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:56:19.622161 5885 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 09:56:19.622718 5877 server_base.cc:1034] running on GCE node
I20260924 09:56:19.623445 5877 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 09:56:19.624714 5877 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 09:56:19.625926 5877 hybrid_clock.cc:648] HybridClock initialized: now 1790243779625881 us; error 36 us; skew 500 ppm
I20260924 09:56:19.626231 5877 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 09:56:19.628523 5877 webserver.cc:460] Webserver started at http://127.0.97.62:37393/ using document root <none> and password file <none>
I20260924 09:56:19.629166 5877 fs_manager.cc:359] Metadata directory not provided
I20260924 09:56:19.629262 5877 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 09:56:19.629547 5877 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 09:56:19.632277 5877 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790243735634835-388-0/minicluster-data/master-0/data/instance:
uuid: "8a16f420c77740ce9b09c1c72492a2e5"
format_stamp: "Formatted at 2026-09-24 09:56:19 on dist-test-slave-w8d2"
server_key: "29193f0012c0fd0d1158e33acb04b189"
server_key_iv: "e0a0847de5e9878bba487d9fcbf0c824"
server_key_version: "encryptionkey@0"
I20260924 09:56:19.633006 5877 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790243735634835-388-0/minicluster-data/master-0/wal/instance:
uuid: "8a16f420c77740ce9b09c1c72492a2e5"
format_stamp: "Formatted at 2026-09-24 09:56:19 on dist-test-slave-w8d2"
server_key: "29193f0012c0fd0d1158e33acb04b189"
server_key_iv: "e0a0847de5e9878bba487d9fcbf0c824"
server_key_version: "encryptionkey@0"
I20260924 09:56:19.637748 5877 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 09:56:19.640862 5891 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 09:56:19.642071 5877 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 09:56:19.642218 5877 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790243735634835-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790243735634835-388-0/minicluster-data/master-0/wal
uuid: "8a16f420c77740ce9b09c1c72492a2e5"
format_stamp: "Formatted at 2026-09-24 09:56:19 on dist-test-slave-w8d2"
server_key: "29193f0012c0fd0d1158e33acb04b189"
server_key_iv: "e0a0847de5e9878bba487d9fcbf0c824"
server_key_version: "encryptionkey@0"
I20260924 09:56:19.642390 5877 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790243735634835-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790243735634835-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790243735634835-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 09:56:19.676033 5877 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 09:56:19.676863 5877 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 09:56:19.677117 5877 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 09:56:19.693465 5877 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:34373
I20260924 09:56:19.693485 5942 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:34373 every 8 connection(s)
I20260924 09:56:19.694713 5877 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790243735634835-388-0/minicluster-data/master-0/data/info.pb
I20260924 09:56:19.696151 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu as pid 5877
I20260924 09:56:19.696352 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790243735634835-388-0/minicluster-data/master-0/wal/instance
I20260924 09:56:19.697126 388 external_mini_cluster.cc:1428] Setting key 0333152a38ead7273b72c910e12e9ba3
I20260924 09:56:19.699110 5943 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 09:56:19.709918 5946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:19.698902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53188 (local address 127.0.97.62:34373)
0924 09:56:19.699834 (+ 932us) server_negotiation.cc:206] Beginning negotiation
0924 09:56:19.699844 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 09:56:19.699890 (+ 46us) server_negotiation.cc:407] Connection header received
0924 09:56:19.701392 (+ 1502us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:56:19.701413 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:56:19.701870 (+ 457us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:56:19.702192 (+ 322us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:56:19.702744 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:19.704128 (+ 1384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:19.705901 (+ 1773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:19.706511 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:19.706627 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:19.707815 (+ 1188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:56:19.707822 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:56:19.707837 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 09:56:19.708357 (+ 520us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:56:19.708437 (+ 80us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 09:56:19.708457 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:56:19.708542 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0924 09:56:19.708741 (+ 199us) server_negotiation.cc:299] Negotiation successful
0924 09:56:19.709043 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":547,"thread_start_us":333,"threads_started":1}
I20260924 09:56:19.711000 5943 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a16f420c77740ce9b09c1c72492a2e5: Bootstrap starting.
I20260924 09:56:19.713397 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 09:56:19.714059 5943 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8a16f420c77740ce9b09c1c72492a2e5: Neither blocks nor log segments found. Creating new log.
I20260924 09:56:19.714985 5943 log.cc:824] T 00000000000000000000000000000000 P 8a16f420c77740ce9b09c1c72492a2e5: Log is configured to *not* fsync() on all Append() calls
I20260924 09:56:19.717550 5943 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a16f420c77740ce9b09c1c72492a2e5: No bootstrap required, opened a new log
I20260924 09:56:19.724383 5943 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8a16f420c77740ce9b09c1c72492a2e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a16f420c77740ce9b09c1c72492a2e5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34373 } }
I20260924 09:56:19.724742 5943 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8a16f420c77740ce9b09c1c72492a2e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 09:56:19.724857 5943 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8a16f420c77740ce9b09c1c72492a2e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a16f420c77740ce9b09c1c72492a2e5, State: Initialized, Role: FOLLOWER
I20260924 09:56:19.725430 5943 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8a16f420c77740ce9b09c1c72492a2e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a16f420c77740ce9b09c1c72492a2e5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34373 } }
I20260924 09:56:19.725643 5943 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8a16f420c77740ce9b09c1c72492a2e5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 09:56:19.725756 5943 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8a16f420c77740ce9b09c1c72492a2e5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 09:56:19.725903 5943 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8a16f420c77740ce9b09c1c72492a2e5 [term 0 FOLLOWER]: Advancing to term 1
I20260924 09:56:19.727911 5943 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8a16f420c77740ce9b09c1c72492a2e5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a16f420c77740ce9b09c1c72492a2e5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34373 } }
I20260924 09:56:19.728344 5943 leader_election.cc:302] T 00000000000000000000000000000000 P 8a16f420c77740ce9b09c1c72492a2e5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8a16f420c77740ce9b09c1c72492a2e5; no voters:
I20260924 09:56:19.728840 5943 leader_election.cc:288] T 00000000000000000000000000000000 P 8a16f420c77740ce9b09c1c72492a2e5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 09:56:19.728955 5948 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8a16f420c77740ce9b09c1c72492a2e5 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 09:56:19.729557 5948 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8a16f420c77740ce9b09c1c72492a2e5 [term 1 LEADER]: Becoming Leader. State: Replica: 8a16f420c77740ce9b09c1c72492a2e5, State: Running, Role: LEADER
I20260924 09:56:19.730016 5948 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8a16f420c77740ce9b09c1c72492a2e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a16f420c77740ce9b09c1c72492a2e5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34373 } }
I20260924 09:56:19.730602 5943 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8a16f420c77740ce9b09c1c72492a2e5 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 09:56:19.732547 5949 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a16f420c77740ce9b09c1c72492a2e5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8a16f420c77740ce9b09c1c72492a2e5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a16f420c77740ce9b09c1c72492a2e5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34373 } } }
I20260924 09:56:19.732915 5949 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a16f420c77740ce9b09c1c72492a2e5 [sys.catalog]: This master's current role is: LEADER
I20260924 09:56:19.733549 5950 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a16f420c77740ce9b09c1c72492a2e5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8a16f420c77740ce9b09c1c72492a2e5. Latest consensus state: current_term: 1 leader_uuid: "8a16f420c77740ce9b09c1c72492a2e5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a16f420c77740ce9b09c1c72492a2e5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34373 } } }
I20260924 09:56:19.733913 5950 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a16f420c77740ce9b09c1c72492a2e5 [sys.catalog]: This master's current role is: LEADER
I20260924 09:56:19.734457 5957 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 09:56:19.739207 5957 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 09:56:19.747267 5957 catalog_manager.cc:1347] Generated new cluster ID: 06223a6bbe4247cb91a9280939b0e6c9
I20260924 09:56:19.747367 5957 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 09:56:19.774524 5957 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 09:56:19.775763 5957 catalog_manager.cc:1504] Loading token signing keys...
I20260924 09:56:19.783620 5957 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8a16f420c77740ce9b09c1c72492a2e5: Generated new TSK 0
I20260924 09:56:19.784418 5957 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 09:56:19.859467 5946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:19.804067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53202 (local address 127.0.97.62:34373)
0924 09:56:19.804249 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0924 09:56:19.804255 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 09:56:19.804542 (+ 287us) server_negotiation.cc:407] Connection header received
0924 09:56:19.804735 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:56:19.804744 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:56:19.804835 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:56:19.804996 (+ 161us) server_negotiation.cc:226] Negotiated authn=JWT
0924 09:56:19.805394 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:19.806502 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:19.807916 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:19.808600 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:19.808741 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:19.809218 (+ 477us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 09:56:19.858705 (+ 49487us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0924 09:56:19.859139 (+ 434us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:53202: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":69}
W20260924 09:56:19.859609 5946 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:53202: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260924 09:56:19.859602 5972 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 09:56:19.803922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34373 (local address 127.0.0.1:53202)
0924 09:56:19.804450 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0924 09:56:19.804466 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 09:56:19.804580 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:56:19.805025 (+ 445us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:56:19.805028 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:56:19.805038 (+ 10us) client_negotiation.cc:189] Negotiated authn=JWT
0924 09:56:19.805259 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:19.805266 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:19.806666 (+ 1400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:19.806671 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:19.807782 (+ 1111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:19.807794 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:19.808755 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:19.808803 (+ 48us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:19.808930 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:19.808963 (+ 33us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 09:56:19.859026 (+ 50063us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0924 09:56:19.859295 (+ 269us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.97.62:34373: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":272,"threads_started":1}
I20260924 09:56:19.860867 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu pid 5877
I20260924 09:56:19.861938 5938 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 09:56:19.957178 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu with pid 5877
2026-09-24T09:56:19Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (720 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (380 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (379 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790243735634835-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 09:56:20 dist-test-slave-w8d2 krb5kdc[5987](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 09:56:20 dist-test-slave-w8d2 krb5kdc[5987](info): set up 2 sockets
Sep 24 09:56:20 dist-test-slave-w8d2 krb5kdc[5987](info): commencing operation
krb5kdc: starting...
W20260924 09:56:21.703635 388 mini_kdc.cc:121] Time spent starting KDC: real 0.954s user 0.003s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790243735634835-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790243735634835-388-0/krb5kdc/test-user.keytab.
Sep 24 09:56:21 dist-test-slave-w8d2 krb5kdc[5987](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243781, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T09:56:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T09:56:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790243735634835-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790243735634835-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260924 09:56:21.881152 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
/tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790243735634835-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790243735634835-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790243735634835-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790243735634835-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46521
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790243735634835-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:46521
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790243735634835-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790243735634835-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790243735634835-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 09:56:22.180920 6000 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 09:56:22.181196 6000 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 09:56:22.181291 6000 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 09:56:22.188313 6000 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 09:56:22.188421 6000 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 09:56:22.188469 6000 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 09:56:22.188522 6000 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 09:56:22.188565 6000 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 09:56:22.198103 6000 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790243735634835-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790243735634835-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:46521
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790243735634835-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46521
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790243735634835-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790243735634835-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 09:56:22.199363 6000 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 09:56:22.200608 6000 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 09:56:22.211584 6005 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:56:22.211580 6008 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:56:22.211649 6006 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 09:56:22.212354 6000 server_base.cc:1034] running on GCE node
I20260924 09:56:22.213083 6000 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 09:56:22.214370 6000 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 09:56:22.215559 6000 hybrid_clock.cc:648] HybridClock initialized: now 1790243782215509 us; error 40 us; skew 500 ppm
I20260924 09:56:22.215881 6000 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 09:56:22 dist-test-slave-w8d2 krb5kdc[5987](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243782, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 09:56:22.221807 6000 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260924 09:56:22.223212 6000 webserver.cc:460] Webserver started at http://127.0.97.62:32961/ using document root <none> and password file <none>
I20260924 09:56:22.223896 6000 fs_manager.cc:359] Metadata directory not provided
I20260924 09:56:22.224002 6000 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 09:56:22.224283 6000 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 09:56:22.227217 6000 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790243735634835-388-0/minicluster-data/master-0/data/instance:
uuid: "10d16c8b01a44305a849431478561384"
format_stamp: "Formatted at 2026-09-24 09:56:22 on dist-test-slave-w8d2"
server_key: "ea2627e09961bb7db487652ec09959c1"
server_key_iv: "56042a6f082f25f9941e2ef1a5bd811b"
server_key_version: "encryptionkey@0"
I20260924 09:56:22.227998 6000 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790243735634835-388-0/minicluster-data/master-0/wal/instance:
uuid: "10d16c8b01a44305a849431478561384"
format_stamp: "Formatted at 2026-09-24 09:56:22 on dist-test-slave-w8d2"
server_key: "ea2627e09961bb7db487652ec09959c1"
server_key_iv: "56042a6f082f25f9941e2ef1a5bd811b"
server_key_version: "encryptionkey@0"
I20260924 09:56:22.232846 6000 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20260924 09:56:22.236020 6015 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 09:56:22.237255 6000 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 09:56:22.237406 6000 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790243735634835-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790243735634835-388-0/minicluster-data/master-0/wal
uuid: "10d16c8b01a44305a849431478561384"
format_stamp: "Formatted at 2026-09-24 09:56:22 on dist-test-slave-w8d2"
server_key: "ea2627e09961bb7db487652ec09959c1"
server_key_iv: "56042a6f082f25f9941e2ef1a5bd811b"
server_key_version: "encryptionkey@0"
I20260924 09:56:22.237571 6000 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790243735634835-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790243735634835-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790243735634835-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 09:56:22.252314 6000 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 09:56:22.255972 6000 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 09:56:22.256223 6000 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 09:56:22.272313 6000 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:46521
I20260924 09:56:22.272344 6066 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:46521 every 8 connection(s)
I20260924 09:56:22.273590 6000 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790243735634835-388-0/minicluster-data/master-0/data/info.pb
I20260924 09:56:22.276170 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu as pid 6000
I20260924 09:56:22.276363 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790243735634835-388-0/minicluster-data/master-0/wal/instance
I20260924 09:56:22.276932 388 external_mini_cluster.cc:1428] Setting key c00c0dcab34b91579ead4f04eab373eb
I20260924 09:56:22.277168 6067 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 09:56:22.290616 6067 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 10d16c8b01a44305a849431478561384: Bootstrap starting.
Sep 24 09:56:22 dist-test-slave-w8d2 krb5kdc[5987](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243781, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 09:56:22.294298 6067 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 10d16c8b01a44305a849431478561384: Neither blocks nor log segments found. Creating new log.
I20260924 09:56:22.295622 6067 log.cc:824] T 00000000000000000000000000000000 P 10d16c8b01a44305a849431478561384: Log is configured to *not* fsync() on all Append() calls
I20260924 09:56:22.298717 6067 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 10d16c8b01a44305a849431478561384: No bootstrap required, opened a new log
I20260924 09:56:22.299698 6070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:22.278698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51658 (local address 127.0.97.62:46521)
0924 09:56:22.279478 (+ 780us) server_negotiation.cc:206] Beginning negotiation
0924 09:56:22.279493 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 09:56:22.279542 (+ 49us) server_negotiation.cc:407] Connection header received
0924 09:56:22.281675 (+ 2133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:56:22.281702 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:56:22.282202 (+ 500us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:56:22.282625 (+ 423us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:56:22.283875 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:22.285779 (+ 1904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:22.286939 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:22.288028 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:22.288208 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:22.291844 (+ 3636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:56:22.291854 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:56:22.291873 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:56:22.291922 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:56:22.295540 (+ 3618us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:56:22.296171 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:56:22.296180 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:56:22.296188 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:56:22.296360 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:56:22.296772 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:56:22.296776 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:56:22.296779 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:56:22.297407 (+ 628us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:56:22.297655 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0924 09:56:22.297994 (+ 339us) server_negotiation.cc:299] Negotiation successful
0924 09:56:22.298390 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":553,"thread_start_us":308,"threads_started":1}
I20260924 09:56:22.306433 6067 raft_consensus.cc:348] T 00000000000000000000000000000000 P 10d16c8b01a44305a849431478561384 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10d16c8b01a44305a849431478561384" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46521 } }
I20260924 09:56:22.306747 6067 raft_consensus.cc:374] T 00000000000000000000000000000000 P 10d16c8b01a44305a849431478561384 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 09:56:22.306815 6067 raft_consensus.cc:729] T 00000000000000000000000000000000 P 10d16c8b01a44305a849431478561384 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10d16c8b01a44305a849431478561384, State: Initialized, Role: FOLLOWER
I20260924 09:56:22.307350 6067 consensus_queue.cc:260] T 00000000000000000000000000000000 P 10d16c8b01a44305a849431478561384 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10d16c8b01a44305a849431478561384" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46521 } }
I20260924 09:56:22.307536 6067 raft_consensus.cc:388] T 00000000000000000000000000000000 P 10d16c8b01a44305a849431478561384 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 09:56:22.307628 6067 raft_consensus.cc:482] T 00000000000000000000000000000000 P 10d16c8b01a44305a849431478561384 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 09:56:22.307758 6067 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 10d16c8b01a44305a849431478561384 [term 0 FOLLOWER]: Advancing to term 1
I20260924 09:56:22.309944 6067 raft_consensus.cc:504] T 00000000000000000000000000000000 P 10d16c8b01a44305a849431478561384 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10d16c8b01a44305a849431478561384" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46521 } }
I20260924 09:56:22.310359 6067 leader_election.cc:302] T 00000000000000000000000000000000 P 10d16c8b01a44305a849431478561384 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 10d16c8b01a44305a849431478561384; no voters:
I20260924 09:56:22.310822 6067 leader_election.cc:288] T 00000000000000000000000000000000 P 10d16c8b01a44305a849431478561384 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 09:56:22.310932 6072 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 10d16c8b01a44305a849431478561384 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 09:56:22.311518 6072 raft_consensus.cc:686] T 00000000000000000000000000000000 P 10d16c8b01a44305a849431478561384 [term 1 LEADER]: Becoming Leader. State: Replica: 10d16c8b01a44305a849431478561384, State: Running, Role: LEADER
I20260924 09:56:22.311955 6072 consensus_queue.cc:237] T 00000000000000000000000000000000 P 10d16c8b01a44305a849431478561384 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10d16c8b01a44305a849431478561384" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46521 } }
I20260924 09:56:22.312616 6067 sys_catalog.cc:564] T 00000000000000000000000000000000 P 10d16c8b01a44305a849431478561384 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 09:56:22.314524 6074 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10d16c8b01a44305a849431478561384 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 10d16c8b01a44305a849431478561384. Latest consensus state: current_term: 1 leader_uuid: "10d16c8b01a44305a849431478561384" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10d16c8b01a44305a849431478561384" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46521 } } }
I20260924 09:56:22.314473 6073 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10d16c8b01a44305a849431478561384 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "10d16c8b01a44305a849431478561384" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10d16c8b01a44305a849431478561384" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46521 } } }
I20260924 09:56:22.314886 6073 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10d16c8b01a44305a849431478561384 [sys.catalog]: This master's current role is: LEADER
I20260924 09:56:22.314886 6074 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10d16c8b01a44305a849431478561384 [sys.catalog]: This master's current role is: LEADER
I20260924 09:56:22.315649 6080 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 09:56:22.320180 6080 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 09:56:22.328498 6080 catalog_manager.cc:1347] Generated new cluster ID: f5887089a1444ffe9981e81e431984ca
I20260924 09:56:22.328608 6080 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 09:56:22.343847 6080 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 09:56:22.345659 6080 catalog_manager.cc:1504] Loading token signing keys...
I20260924 09:56:22.354327 6080 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 10d16c8b01a44305a849431478561384: Generated new TSK 0
I20260924 09:56:22.355396 6080 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790243735634835-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790243735634835-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260924 09:56:22.402712 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
/tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790243735634835-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790243735634835-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790243735634835-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790243735634835-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46521
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790243735634835-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790243735634835-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790243735634835-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790243735634835-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 09:56:22.709616 6093 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 09:56:22.709887 6093 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 09:56:22.710014 6093 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 09:56:22.717145 6093 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 09:56:22.717254 6093 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 09:56:22.717401 6093 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260924 09:56:22.727136 6093 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790243735634835-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790243735634835-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790243735634835-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790243735634835-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46521
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790243735634835-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 09:56:22.728514 6093 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 09:56:22.729789 6093 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 09:56:22.741840 6098 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:56:22.742587 6099 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:56:22.742636 6101 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 09:56:22.743175 6093 server_base.cc:1034] running on GCE node
I20260924 09:56:22.743942 6093 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 09:56:22.745800 6093 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 09:56:22.747052 6093 hybrid_clock.cc:648] HybridClock initialized: now 1790243782746984 us; error 60 us; skew 500 ppm
I20260924 09:56:22.747457 6093 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 09:56:22 dist-test-slave-w8d2 krb5kdc[5987](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243782, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 09:56:22.750927 6093 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260924 09:56:22.752362 6093 webserver.cc:460] Webserver started at http://127.0.97.1:42657/ using document root <none> and password file <none>
I20260924 09:56:22.753036 6093 fs_manager.cc:359] Metadata directory not provided
I20260924 09:56:22.753145 6093 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 09:56:22.753412 6093 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 09:56:22.756251 6093 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790243735634835-388-0/minicluster-data/ts-0/data/instance:
uuid: "e9c82eb81ad14679bcbd92c68b9e0a82"
format_stamp: "Formatted at 2026-09-24 09:56:22 on dist-test-slave-w8d2"
server_key: "14712912430e9232f0e9f565d607dc24"
server_key_iv: "619f3a1fb654d1bdb4df8ecf3672719c"
server_key_version: "encryptionkey@0"
I20260924 09:56:22.756983 6093 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790243735634835-388-0/minicluster-data/ts-0/wal/instance:
uuid: "e9c82eb81ad14679bcbd92c68b9e0a82"
format_stamp: "Formatted at 2026-09-24 09:56:22 on dist-test-slave-w8d2"
server_key: "14712912430e9232f0e9f565d607dc24"
server_key_iv: "619f3a1fb654d1bdb4df8ecf3672719c"
server_key_version: "encryptionkey@0"
I20260924 09:56:22.761997 6093 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 09:56:22.765291 6108 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 09:56:22.766642 6093 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 09:56:22.766798 6093 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790243735634835-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790243735634835-388-0/minicluster-data/ts-0/wal
uuid: "e9c82eb81ad14679bcbd92c68b9e0a82"
format_stamp: "Formatted at 2026-09-24 09:56:22 on dist-test-slave-w8d2"
server_key: "14712912430e9232f0e9f565d607dc24"
server_key_iv: "619f3a1fb654d1bdb4df8ecf3672719c"
server_key_version: "encryptionkey@0"
I20260924 09:56:22.766974 6093 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790243735634835-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790243735634835-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790243735634835-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 09:56:22.791893 6093 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 09:56:22.795933 6093 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 09:56:22.796214 6093 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 09:56:22.797163 6093 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 09:56:22.798483 6093 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 09:56:22.798561 6093 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 09:56:22.798674 6093 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 09:56:22.798732 6093 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 09:56:22.827078 6093 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:37381
I20260924 09:56:22.827119 6220 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:37381 every 8 connection(s)
I20260924 09:56:22.828346 6093 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790243735634835-388-0/minicluster-data/ts-0/data/info.pb
I20260924 09:56:22.830688 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu as pid 6093
I20260924 09:56:22.830891 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790243735634835-388-0/minicluster-data/ts-0/wal/instance
I20260924 09:56:22.831606 388 external_mini_cluster.cc:1428] Setting key 3e5b03386924b818dac3df4ffc2df60e
Sep 24 09:56:22 dist-test-slave-w8d2 krb5kdc[5987](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243782, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 09:56:22.848798 6223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:22.830780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:47073 (local address 127.0.97.62:46521)
0924 09:56:22.831217 (+ 437us) server_negotiation.cc:206] Beginning negotiation
0924 09:56:22.831223 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 09:56:22.832277 (+ 1054us) server_negotiation.cc:407] Connection header received
0924 09:56:22.834317 (+ 2040us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:56:22.834321 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:56:22.834379 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:56:22.834457 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:56:22.836330 (+ 1873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:22.837045 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:22.839028 (+ 1983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:22.839428 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:22.839529 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:22.842936 (+ 3407us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:56:22.842942 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:56:22.842947 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:56:22.842983 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:56:22.845853 (+ 2870us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:56:22.846523 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:56:22.846527 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:56:22.846531 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:56:22.846663 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:56:22.847036 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:56:22.847041 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:56:22.847045 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:56:22.847410 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:56:22.847547 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0924 09:56:22.848191 (+ 644us) server_negotiation.cc:299] Negotiation successful
0924 09:56:22.848435 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":202,"threads_started":1}
I20260924 09:56:22.849607 6224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:22.831324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46521 (local address 127.0.97.1:47073)
0924 09:56:22.832103 (+ 779us) negotiation.cc:107] Waiting for socket to connect
0924 09:56:22.832153 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0924 09:56:22.834033 (+ 1880us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:56:22.834726 (+ 693us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:56:22.834738 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:56:22.835445 (+ 707us) client_negotiation.cc:189] Negotiated authn=SASL
0924 09:56:22.836190 (+ 745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:22.836213 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:22.837170 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:22.837174 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:22.838912 (+ 1738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:22.838925 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:22.839614 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:22.839619 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:22.839828 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:22.840440 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 09:56:22.840461 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 09:56:22.842696 (+ 2235us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 09:56:22.845999 (+ 3303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 09:56:22.846005 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 09:56:22.846018 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 09:56:22.846424 (+ 406us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 09:56:22.846751 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 09:56:22.846754 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 09:56:22.846756 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 09:56:22.846925 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 09:56:22.847523 (+ 598us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 09:56:22.847529 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 09:56:22.847849 (+ 320us) client_negotiation.cc:769] Sending connection context
0924 09:56:22.848077 (+ 228us) client_negotiation.cc:240] Negotiation successful
0924 09:56:22.848499 (+ 422us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":513,"thread_start_us":315,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260924 09:56:22.851101 6221 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46521
I20260924 09:56:22.851404 6221 heartbeater.cc:464] Registering TS with master...
I20260924 09:56:22.852066 6221 heartbeater.cc:511] Master 127.0.97.62:46521 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260924 09:56:22.854120 6032 ts_manager.cc:194] Registered new tserver with Master: e9c82eb81ad14679bcbd92c68b9e0a82 (127.0.97.1:37381)
I20260924 09:56:22.855624 6032 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:47073
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790243735634835-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790243735634835-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260924 09:56:22.874384 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
/tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790243735634835-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790243735634835-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790243735634835-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790243735634835-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46521
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790243735634835-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790243735634835-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790243735634835-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790243735634835-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 09:56:23.184010 6227 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 09:56:23.184296 6227 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 09:56:23.184397 6227 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 09:56:23.191524 6227 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 09:56:23.191633 6227 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 09:56:23.191751 6227 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260924 09:56:23.201198 6227 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790243735634835-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790243735634835-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790243735634835-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790243735634835-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46521
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790243735634835-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 09:56:23.202401 6227 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 09:56:23.203687 6227 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 09:56:23.215903 6233 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:56:23.215939 6235 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:56:23.216099 6232 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 09:56:23.216691 6227 server_base.cc:1034] running on GCE node
I20260924 09:56:23.217240 6227 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 09:56:23.218065 6227 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 09:56:23.219269 6227 hybrid_clock.cc:648] HybridClock initialized: now 1790243783219200 us; error 45 us; skew 500 ppm
I20260924 09:56:23.219568 6227 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 09:56:23 dist-test-slave-w8d2 krb5kdc[5987](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243783, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 09:56:23.226737 6227 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260924 09:56:23.228240 6227 webserver.cc:460] Webserver started at http://127.0.97.2:38219/ using document root <none> and password file <none>
I20260924 09:56:23.228888 6227 fs_manager.cc:359] Metadata directory not provided
I20260924 09:56:23.229007 6227 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 09:56:23.229274 6227 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 09:56:23.232080 6227 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790243735634835-388-0/minicluster-data/ts-1/data/instance:
uuid: "eb24527a68ce458eb3e1e254eac4547f"
format_stamp: "Formatted at 2026-09-24 09:56:23 on dist-test-slave-w8d2"
server_key: "319d3cb822be3af7484bb66071f34477"
server_key_iv: "157e11936a803879e3254287ba7b180c"
server_key_version: "encryptionkey@0"
I20260924 09:56:23.232777 6227 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790243735634835-388-0/minicluster-data/ts-1/wal/instance:
uuid: "eb24527a68ce458eb3e1e254eac4547f"
format_stamp: "Formatted at 2026-09-24 09:56:23 on dist-test-slave-w8d2"
server_key: "319d3cb822be3af7484bb66071f34477"
server_key_iv: "157e11936a803879e3254287ba7b180c"
server_key_version: "encryptionkey@0"
I20260924 09:56:23.237560 6227 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260924 09:56:23.240756 6242 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 09:56:23.242017 6227 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 09:56:23.242156 6227 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790243735634835-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790243735634835-388-0/minicluster-data/ts-1/wal
uuid: "eb24527a68ce458eb3e1e254eac4547f"
format_stamp: "Formatted at 2026-09-24 09:56:23 on dist-test-slave-w8d2"
server_key: "319d3cb822be3af7484bb66071f34477"
server_key_iv: "157e11936a803879e3254287ba7b180c"
server_key_version: "encryptionkey@0"
I20260924 09:56:23.242319 6227 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790243735634835-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790243735634835-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790243735634835-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 09:56:23.262079 6227 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 09:56:23.265864 6227 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 09:56:23.266098 6227 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 09:56:23.266947 6227 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 09:56:23.268183 6227 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 09:56:23.268252 6227 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 09:56:23.268342 6227 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 09:56:23.268384 6227 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 09:56:23.301589 6227 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:36917
I20260924 09:56:23.301640 6354 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:36917 every 8 connection(s)
I20260924 09:56:23.302802 6227 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790243735634835-388-0/minicluster-data/ts-1/data/info.pb
I20260924 09:56:23.311667 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu as pid 6227
I20260924 09:56:23.311830 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790243735634835-388-0/minicluster-data/ts-1/wal/instance
I20260924 09:56:23.312348 388 external_mini_cluster.cc:1428] Setting key 1bb71692089410dd62619c4a5bd96e5d
Sep 24 09:56:23 dist-test-slave-w8d2 krb5kdc[5987](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243783, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 09:56:23.322247 6223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:23.305189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:50315 (local address 127.0.97.62:46521)
0924 09:56:23.305396 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0924 09:56:23.305402 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 09:56:23.306606 (+ 1204us) server_negotiation.cc:407] Connection header received
0924 09:56:23.308301 (+ 1695us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:56:23.308305 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:56:23.308372 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:56:23.308454 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:56:23.309814 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:23.310508 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:23.312098 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:23.312671 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:23.312820 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:23.316683 (+ 3863us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:56:23.316687 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:56:23.316691 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:56:23.316722 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:56:23.319079 (+ 2357us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:56:23.319985 (+ 906us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:56:23.319991 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:56:23.319995 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:56:23.320149 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:56:23.320557 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:56:23.320561 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:56:23.320564 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:56:23.320910 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:56:23.321040 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0924 09:56:23.321714 (+ 674us) server_negotiation.cc:299] Negotiation successful
0924 09:56:23.321934 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260924 09:56:23.323062 6357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:23.305748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46521 (local address 127.0.97.2:50315)
0924 09:56:23.306455 (+ 707us) negotiation.cc:107] Waiting for socket to connect
0924 09:56:23.306493 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 09:56:23.308069 (+ 1576us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:56:23.308637 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:56:23.308645 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:56:23.309131 (+ 486us) client_negotiation.cc:189] Negotiated authn=SASL
0924 09:56:23.309684 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:23.309701 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:23.310620 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:23.310623 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:23.311965 (+ 1342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:23.311976 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:23.312946 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:23.312954 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:23.313231 (+ 277us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:23.313911 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 09:56:23.313945 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0924 09:56:23.316459 (+ 2514us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 09:56:23.319216 (+ 2757us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 09:56:23.319225 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 09:56:23.319275 (+ 50us) client_negotiation.cc:760] calling sasl_client_step()
0924 09:56:23.319843 (+ 568us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 09:56:23.320244 (+ 401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 09:56:23.320247 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 09:56:23.320250 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 09:56:23.320452 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 09:56:23.321009 (+ 557us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 09:56:23.321016 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 09:56:23.321354 (+ 338us) client_negotiation.cc:769] Sending connection context
0924 09:56:23.321599 (+ 245us) client_negotiation.cc:240] Negotiation successful
0924 09:56:23.321985 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":273,"threads_started":1}
I20260924 09:56:23.324404 6355 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46521
I20260924 09:56:23.324728 6355 heartbeater.cc:464] Registering TS with master...
I20260924 09:56:23.325546 6355 heartbeater.cc:511] Master 127.0.97.62:46521 requested a full tablet report, sending...
I20260924 09:56:23.327056 6032 ts_manager.cc:194] Registered new tserver with Master: eb24527a68ce458eb3e1e254eac4547f (127.0.97.2:36917)
I20260924 09:56:23.327931 6032 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:50315
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790243735634835-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790243735634835-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260924 09:56:23.355826 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
/tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790243735634835-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790243735634835-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790243735634835-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790243735634835-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46521
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790243735634835-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790243735634835-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790243735634835-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790243735634835-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 09:56:23.674237 6360 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 09:56:23.674527 6360 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 09:56:23.674661 6360 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 09:56:23.681859 6360 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 09:56:23.681984 6360 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 09:56:23.682129 6360 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260924 09:56:23.692035 6360 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790243735634835-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790243735634835-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790243735634835-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790243735634835-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46521
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790243735634835-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 09:56:23.693284 6360 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 09:56:23.694568 6360 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 09:56:23.706518 6365 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:56:23.706543 6366 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:56:23.706475 6368 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 09:56:23.707314 6360 server_base.cc:1034] running on GCE node
I20260924 09:56:23.707870 6360 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 09:56:23.708719 6360 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 09:56:23.709901 6360 hybrid_clock.cc:648] HybridClock initialized: now 1790243783709850 us; error 47 us; skew 500 ppm
I20260924 09:56:23.710182 6360 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 09:56:23 dist-test-slave-w8d2 krb5kdc[5987](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243783, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 09:56:23.713073 6360 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260924 09:56:23.714362 6360 webserver.cc:460] Webserver started at http://127.0.97.3:46407/ using document root <none> and password file <none>
I20260924 09:56:23.714986 6360 fs_manager.cc:359] Metadata directory not provided
I20260924 09:56:23.715081 6360 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 09:56:23.715350 6360 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 09:56:23.718007 6360 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790243735634835-388-0/minicluster-data/ts-2/data/instance:
uuid: "91309e297d15406189bef3654880ab49"
format_stamp: "Formatted at 2026-09-24 09:56:23 on dist-test-slave-w8d2"
server_key: "e6364100cdee4502f0efb631da2bb8d8"
server_key_iv: "8612c1588fb457bb736b210e6f4aa171"
server_key_version: "encryptionkey@0"
I20260924 09:56:23.718686 6360 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790243735634835-388-0/minicluster-data/ts-2/wal/instance:
uuid: "91309e297d15406189bef3654880ab49"
format_stamp: "Formatted at 2026-09-24 09:56:23 on dist-test-slave-w8d2"
server_key: "e6364100cdee4502f0efb631da2bb8d8"
server_key_iv: "8612c1588fb457bb736b210e6f4aa171"
server_key_version: "encryptionkey@0"
I20260924 09:56:23.723703 6360 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260924 09:56:23.726886 6375 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 09:56:23.728183 6360 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 09:56:23.728319 6360 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790243735634835-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790243735634835-388-0/minicluster-data/ts-2/wal
uuid: "91309e297d15406189bef3654880ab49"
format_stamp: "Formatted at 2026-09-24 09:56:23 on dist-test-slave-w8d2"
server_key: "e6364100cdee4502f0efb631da2bb8d8"
server_key_iv: "8612c1588fb457bb736b210e6f4aa171"
server_key_version: "encryptionkey@0"
I20260924 09:56:23.728513 6360 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790243735634835-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790243735634835-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790243735634835-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 09:56:23.740723 6360 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 09:56:23.744521 6360 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 09:56:23.744771 6360 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 09:56:23.745678 6360 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 09:56:23.747002 6360 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 09:56:23.747072 6360 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 09:56:23.747175 6360 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 09:56:23.747224 6360 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 09:56:23.775581 6360 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:36455
I20260924 09:56:23.775627 6487 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:36455 every 8 connection(s)
I20260924 09:56:23.776753 6360 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790243735634835-388-0/minicluster-data/ts-2/data/info.pb
I20260924 09:56:23.782246 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu as pid 6360
I20260924 09:56:23.782429 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790243735634835-388-0/minicluster-data/ts-2/wal/instance
I20260924 09:56:23.782824 388 external_mini_cluster.cc:1428] Setting key cc1c6b2ae7c46f28dac59c1bf00192f2
Sep 24 09:56:23 dist-test-slave-w8d2 krb5kdc[5987](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243783, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 09:56:23.794831 6223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:23.779039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:39697 (local address 127.0.97.62:46521)
0924 09:56:23.779194 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0924 09:56:23.779198 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 09:56:23.780367 (+ 1169us) server_negotiation.cc:407] Connection header received
0924 09:56:23.782030 (+ 1663us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:56:23.782036 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:56:23.782119 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:56:23.782229 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:56:23.783798 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:23.784808 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:23.786489 (+ 1681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:23.786860 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:23.786968 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:23.790011 (+ 3043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:56:23.790016 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:56:23.790019 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:56:23.790047 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:56:23.792255 (+ 2208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:56:23.792937 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:56:23.792940 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:56:23.792942 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:56:23.793054 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:56:23.793442 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:56:23.793445 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:56:23.793447 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:56:23.793688 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:56:23.793789 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0924 09:56:23.794430 (+ 641us) server_negotiation.cc:299] Negotiation successful
0924 09:56:23.794591 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 09:56:23.795821 6490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:23.779522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46521 (local address 127.0.97.3:39697)
0924 09:56:23.780208 (+ 686us) negotiation.cc:107] Waiting for socket to connect
0924 09:56:23.780243 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 09:56:23.781760 (+ 1517us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:56:23.782422 (+ 662us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:56:23.782432 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:56:23.782985 (+ 553us) client_negotiation.cc:189] Negotiated authn=SASL
0924 09:56:23.783659 (+ 674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:23.783683 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:23.784950 (+ 1267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:23.784955 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:23.786378 (+ 1423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:23.786389 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:23.787051 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:23.787056 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:23.787266 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:23.787856 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 09:56:23.787899 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0924 09:56:23.789837 (+ 1938us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 09:56:23.792404 (+ 2567us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 09:56:23.792411 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 09:56:23.792425 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 09:56:23.792842 (+ 417us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 09:56:23.793167 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 09:56:23.793171 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 09:56:23.793173 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 09:56:23.793356 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 09:56:23.793775 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 09:56:23.793781 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 09:56:23.794130 (+ 349us) client_negotiation.cc:769] Sending connection context
0924 09:56:23.794355 (+ 225us) client_negotiation.cc:240] Negotiation successful
0924 09:56:23.794748 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":287,"threads_started":1}
I20260924 09:56:23.797191 6488 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46521
I20260924 09:56:23.797505 6488 heartbeater.cc:464] Registering TS with master...
I20260924 09:56:23.798236 6488 heartbeater.cc:511] Master 127.0.97.62:46521 requested a full tablet report, sending...
I20260924 09:56:23.799521 6031 ts_manager.cc:194] Registered new tserver with Master: 91309e297d15406189bef3654880ab49 (127.0.97.3:36455)
I20260924 09:56:23.800315 6031 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:39697
I20260924 09:56:23.809988 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790243735634835-388-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790243735634835-388-0/krb5kdc/test-admin.keytab.
Sep 24 09:56:23 dist-test-slave-w8d2 krb5kdc[5987](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243783, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 09:56:23.833918 388 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260924 09:56:23.838367 6495 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 09:56:23.836296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54606 (local address 127.0.97.1:37381)
0924 09:56:23.836861 (+ 565us) server_negotiation.cc:206] Beginning negotiation
0924 09:56:23.836868 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 09:56:23.836894 (+ 26us) server_negotiation.cc:407] Connection header received
0924 09:56:23.836948 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:56:23.836954 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:56:23.837161 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:56:23.837282 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:56:23.838010 (+ 728us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54606: BlockingRecv error: recv got EOF from 127.0.0.1:54606 (error 108)
Metrics: {"server-negotiator.queue_time_us":433,"thread_start_us":297,"threads_started":1}
W20260924 09:56:23.838506 388 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790243735634835-388-0/krb5kdc/krb5cc' of type FILE
Sep 24 09:56:23 dist-test-slave-w8d2 krb5kdc[5987](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243783, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 09:56:23.840168 388 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 24 09:56:23 dist-test-slave-w8d2 krb5kdc[5987](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243783, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 09:56:23.841452 388 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 24 09:56:23 dist-test-slave-w8d2 krb5kdc[5987](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243783, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260924 09:56:23.855474 6495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:23.843007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54608 (local address 127.0.97.1:37381)
0924 09:56:23.843180 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0924 09:56:23.843184 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 09:56:23.843270 (+ 86us) server_negotiation.cc:407] Connection header received
0924 09:56:23.843444 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:56:23.843448 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:56:23.843502 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:56:23.843578 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:56:23.844469 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:23.845417 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:23.846682 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:23.847219 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:23.847352 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:23.850609 (+ 3257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:56:23.850619 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:56:23.850634 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:56:23.850664 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:56:23.853216 (+ 2552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:56:23.853799 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:56:23.853806 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:56:23.853809 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:56:23.853913 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:56:23.854188 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:56:23.854190 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:56:23.854192 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:56:23.854493 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:56:23.854688 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0924 09:56:23.854928 (+ 240us) server_negotiation.cc:299] Negotiation successful
0924 09:56:23.855152 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 09:56:23.858320 6221 heartbeater.cc:503] Master 127.0.97.62:46521 was elected leader, sending a full tablet report...
W20260924 09:56:23.860819 6495 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 09:56:23.859350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54612 (local address 127.0.97.1:37381)
0924 09:56:23.859513 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0924 09:56:23.859519 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 09:56:23.859674 (+ 155us) server_negotiation.cc:407] Connection header received
0924 09:56:23.859841 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:56:23.859846 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:56:23.859914 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:56:23.859996 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:56:23.860603 (+ 607us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54612: BlockingRecv error: recv got EOF from 127.0.0.1:54612 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260924 09:56:23.860929 388 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790243735634835-388-0/krb5kdc/krb5cc' of type FILE
Sep 24 09:56:23 dist-test-slave-w8d2 krb5kdc[5987](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243783, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 09:56:23.862286 388 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 24 09:56:23 dist-test-slave-w8d2 krb5kdc[5987](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243783, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 09:56:23.863432 388 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 24 09:56:23 dist-test-slave-w8d2 krb5kdc[5987](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243783, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260924 09:56:23.875728 6495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:23.864559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54614 (local address 127.0.97.1:37381)
0924 09:56:23.864704 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0924 09:56:23.864709 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 09:56:23.864996 (+ 287us) server_negotiation.cc:407] Connection header received
0924 09:56:23.865121 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:56:23.865126 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:56:23.865177 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:56:23.865249 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:56:23.866093 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:23.866754 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:23.867957 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:23.868356 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:23.868457 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:23.871736 (+ 3279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:56:23.871741 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:56:23.871744 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:56:23.871772 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:56:23.873865 (+ 2093us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:56:23.874376 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:56:23.874380 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:56:23.874382 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:56:23.874494 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:56:23.874749 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:56:23.874752 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:56:23.874754 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:56:23.874983 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:56:23.875153 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0924 09:56:23.875364 (+ 211us) server_negotiation.cc:299] Negotiation successful
0924 09:56:23.875517 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
W20260924 09:56:23.879391 6495 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 09:56:23.877839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54630 (local address 127.0.97.1:37381)
0924 09:56:23.877978 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0924 09:56:23.877983 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 09:56:23.878237 (+ 254us) server_negotiation.cc:407] Connection header received
0924 09:56:23.878374 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:56:23.878378 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:56:23.878451 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:56:23.878525 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:56:23.879158 (+ 633us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54630: BlockingRecv error: recv got EOF from 127.0.0.1:54630 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260924 09:56:23.879628 388 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790243735634835-388-0/krb5kdc/krb5cc' of type FILE
Sep 24 09:56:23 dist-test-slave-w8d2 krb5kdc[5987](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243783, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 09:56:23.880960 388 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 24 09:56:23 dist-test-slave-w8d2 krb5kdc[5987](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243783, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 09:56:23.882102 388 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 24 09:56:23 dist-test-slave-w8d2 krb5kdc[5987](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243783, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260924 09:56:23.894107 6495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:23.883187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54646 (local address 127.0.97.1:37381)
0924 09:56:23.883381 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0924 09:56:23.883386 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 09:56:23.883565 (+ 179us) server_negotiation.cc:407] Connection header received
0924 09:56:23.883682 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:56:23.883685 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:56:23.883735 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:56:23.883798 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:56:23.884630 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:23.885327 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:23.886439 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:23.886828 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:23.886907 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:23.889861 (+ 2954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:56:23.889865 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:56:23.889868 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:56:23.889896 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:56:23.892054 (+ 2158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:56:23.892636 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:56:23.892639 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:56:23.892642 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:56:23.892769 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:56:23.893038 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:56:23.893041 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:56:23.893042 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:56:23.893275 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:56:23.893481 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0924 09:56:23.893656 (+ 175us) server_negotiation.cc:299] Negotiation successful
0924 09:56:23.893800 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260924 09:56:23.895517 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu pid 6093
I20260924 09:56:23.905970 6495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:23.896114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54658 (local address 127.0.97.1:37381)
0924 09:56:23.896281 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0924 09:56:23.896286 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 09:56:23.896491 (+ 205us) server_negotiation.cc:407] Connection header received
0924 09:56:23.896599 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:56:23.896602 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:56:23.896652 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:56:23.896719 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:56:23.897537 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:23.898169 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:23.899381 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:23.899778 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:23.899861 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:23.901571 (+ 1710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:56:23.901577 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:56:23.901580 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:56:23.901607 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:56:23.903706 (+ 2099us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:56:23.904265 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:56:23.904271 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:56:23.904275 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:56:23.904393 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:56:23.904711 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:56:23.904716 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:56:23.904720 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:56:23.904960 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:56:23.905178 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0924 09:56:23.905479 (+ 301us) server_negotiation.cc:299] Negotiation successful
0924 09:56:23.905666 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 09:56:23.906481 6216 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 09:56:23.986347 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu with pid 6093
I20260924 09:56:23.998775 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu pid 6227
Sep 24 09:56:23 dist-test-slave-w8d2 krb5kdc[5987](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243783, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260924 09:56:24.012724 6508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:23.999540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43978 (local address 127.0.97.2:36917)
0924 09:56:24.000017 (+ 477us) server_negotiation.cc:206] Beginning negotiation
0924 09:56:24.000024 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 09:56:24.000050 (+ 26us) server_negotiation.cc:407] Connection header received
0924 09:56:24.000108 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:56:24.000114 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:56:24.000321 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:56:24.000446 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:56:24.001314 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:24.002316 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:24.003530 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:24.004091 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:24.004187 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:24.007522 (+ 3335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:56:24.007533 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:56:24.007547 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:56:24.007579 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:56:24.010145 (+ 2566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:56:24.010791 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:56:24.010800 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:56:24.010804 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:56:24.010971 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:56:24.011456 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:56:24.011460 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:56:24.011462 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:56:24.011800 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:56:24.012058 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0924 09:56:24.012265 (+ 207us) server_negotiation.cc:299] Negotiation successful
0924 09:56:24.012451 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":229,"threads_started":1}
I20260924 09:56:24.013950 6350 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 09:56:24.093456 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu with pid 6227
I20260924 09:56:24.105906 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu pid 6360
Sep 24 09:56:24 dist-test-slave-w8d2 krb5kdc[5987](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243783, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260924 09:56:24.120018 6510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:24.106701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49170 (local address 127.0.97.3:36455)
0924 09:56:24.107215 (+ 514us) server_negotiation.cc:206] Beginning negotiation
0924 09:56:24.107222 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 09:56:24.107274 (+ 52us) server_negotiation.cc:407] Connection header received
0924 09:56:24.107333 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:56:24.107340 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:56:24.107562 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:56:24.107700 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:56:24.108669 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:24.109658 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:24.110848 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:24.111476 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:24.111593 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:24.115115 (+ 3522us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:56:24.115125 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:56:24.115139 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:56:24.115168 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:56:24.117557 (+ 2389us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:56:24.118152 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:56:24.118158 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:56:24.118161 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:56:24.118268 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:56:24.118593 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:56:24.118597 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:56:24.118601 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:56:24.118901 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:56:24.119177 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0924 09:56:24.119461 (+ 284us) server_negotiation.cc:299] Negotiation successful
0924 09:56:24.119731 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":381,"thread_start_us":236,"threads_started":1}
I20260924 09:56:24.121308 6483 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 09:56:24.203222 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu with pid 6360
I20260924 09:56:24.215391 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu pid 6000
I20260924 09:56:24.216507 6062 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 09:56:24.315542 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu with pid 6000
2026-09-24T09:56:24Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (3604 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790243735634835-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 09:56:24 dist-test-slave-w8d2 krb5kdc[6515](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 09:56:24 dist-test-slave-w8d2 krb5kdc[6515](info): set up 2 sockets
Sep 24 09:56:24 dist-test-slave-w8d2 krb5kdc[6515](info): commencing operation
krb5kdc: starting...
W20260924 09:56:25.315712 388 mini_kdc.cc:121] Time spent starting KDC: real 0.963s user 0.005s sys 0.012s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790243735634835-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790243735634835-388-0/krb5kdc/test-user.keytab.
Sep 24 09:56:25 dist-test-slave-w8d2 krb5kdc[6515](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243785, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T09:56:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T09:56:25Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790243735634835-388-0/krb5kdc/oryx_127.0.97.62.keytab.
Entry for principal oryx/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790243735634835-388-0/krb5kdc/oryx_127.0.97.62.keytab.
I20260924 09:56:25.498656 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
/tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790243735634835-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790243735634835-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790243735634835-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790243735634835-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43811
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790243735634835-388-0/krb5kdc/oryx_127.0.97.62.keytab
--principal=oryx/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:43811
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790243735634835-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790243735634835-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790243735634835-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 09:56:25.801177 6528 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 09:56:25.801466 6528 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 09:56:25.801606 6528 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 09:56:25.808670 6528 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 09:56:25.808792 6528 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 09:56:25.808866 6528 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 09:56:25.808938 6528 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 09:56:25.808997 6528 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 09:56:25.818506 6528 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790243735634835-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790243735634835-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:43811
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790243735634835-388-0/krb5kdc/oryx_127.0.97.62.keytab
--principal=oryx/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43811
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790243735634835-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790243735634835-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 09:56:25.819825 6528 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 09:56:25.821103 6528 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 09:56:25.832144 6533 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:56:25.832072 6536 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:56:25.832152 6534 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 09:56:25.832784 6528 server_base.cc:1034] running on GCE node
I20260924 09:56:25.833528 6528 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 09:56:25.834875 6528 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 09:56:25.836051 6528 hybrid_clock.cc:648] HybridClock initialized: now 1790243785836012 us; error 33 us; skew 500 ppm
I20260924 09:56:25.836364 6528 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 09:56:25 dist-test-slave-w8d2 krb5kdc[6515](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243785, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 09:56:25.839130 6528 init.cc:377] Logged in from keytab as oryx/127.0.97.62@KRBTEST.COM (short username oryx)
I20260924 09:56:25.840453 6528 webserver.cc:460] Webserver started at http://127.0.97.62:46193/ using document root <none> and password file <none>
I20260924 09:56:25.841162 6528 fs_manager.cc:359] Metadata directory not provided
I20260924 09:56:25.841281 6528 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 09:56:25.841543 6528 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 09:56:25.844401 6528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790243735634835-388-0/minicluster-data/master-0/data/instance:
uuid: "c65adc94d5fe46cb8d1244957f570265"
format_stamp: "Formatted at 2026-09-24 09:56:25 on dist-test-slave-w8d2"
server_key: "3360842acdc2051696b1048a0c09a424"
server_key_iv: "40d291b0cadd63e8629c7845f5badd8e"
server_key_version: "encryptionkey@0"
I20260924 09:56:25.845093 6528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790243735634835-388-0/minicluster-data/master-0/wal/instance:
uuid: "c65adc94d5fe46cb8d1244957f570265"
format_stamp: "Formatted at 2026-09-24 09:56:25 on dist-test-slave-w8d2"
server_key: "3360842acdc2051696b1048a0c09a424"
server_key_iv: "40d291b0cadd63e8629c7845f5badd8e"
server_key_version: "encryptionkey@0"
I20260924 09:56:25.849836 6528 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260924 09:56:25.853101 6543 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 09:56:25.854337 6528 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 09:56:25.854476 6528 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790243735634835-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790243735634835-388-0/minicluster-data/master-0/wal
uuid: "c65adc94d5fe46cb8d1244957f570265"
format_stamp: "Formatted at 2026-09-24 09:56:25 on dist-test-slave-w8d2"
server_key: "3360842acdc2051696b1048a0c09a424"
server_key_iv: "40d291b0cadd63e8629c7845f5badd8e"
server_key_version: "encryptionkey@0"
I20260924 09:56:25.854633 6528 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790243735634835-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790243735634835-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790243735634835-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 09:56:25.879436 6528 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 09:56:25.883322 6528 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 09:56:25.883558 6528 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 09:56:25.899031 6528 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:43811
I20260924 09:56:25.899058 6594 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:43811 every 8 connection(s)
I20260924 09:56:25.900265 6528 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790243735634835-388-0/minicluster-data/master-0/data/info.pb
I20260924 09:56:25.903757 6595 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 09:56:25.903784 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu as pid 6528
I20260924 09:56:25.903991 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790243735634835-388-0/minicluster-data/master-0/wal/instance
I20260924 09:56:25.904561 388 external_mini_cluster.cc:1428] Setting key 194aae00e7e82f3cbc9b2ea026238e0e
I20260924 09:56:25.917008 6595 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c65adc94d5fe46cb8d1244957f570265: Bootstrap starting.
Sep 24 09:56:25 dist-test-slave-w8d2 krb5kdc[6515](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243785, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260924 09:56:25.919994 6595 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c65adc94d5fe46cb8d1244957f570265: Neither blocks nor log segments found. Creating new log.
I20260924 09:56:25.921232 6595 log.cc:824] T 00000000000000000000000000000000 P c65adc94d5fe46cb8d1244957f570265: Log is configured to *not* fsync() on all Append() calls
I20260924 09:56:25.924258 6595 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c65adc94d5fe46cb8d1244957f570265: No bootstrap required, opened a new log
I20260924 09:56:25.926934 6598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:25.906319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52572 (local address 127.0.97.62:43811)
0924 09:56:25.907909 (+ 1590us) server_negotiation.cc:206] Beginning negotiation
0924 09:56:25.907923 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 09:56:25.907973 (+ 50us) server_negotiation.cc:407] Connection header received
0924 09:56:25.909970 (+ 1997us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:56:25.909996 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:56:25.910465 (+ 469us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:56:25.910894 (+ 429us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:56:25.912027 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:25.913897 (+ 1870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:25.915068 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:25.916054 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:25.916210 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:25.919560 (+ 3350us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:56:25.919570 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:56:25.919588 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:56:25.919634 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:56:25.922901 (+ 3267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:56:25.923513 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:56:25.923521 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:56:25.923528 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:56:25.923738 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:56:25.924048 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:56:25.924052 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:56:25.924055 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:56:25.924687 (+ 632us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:56:25.924966 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0924 09:56:25.925250 (+ 284us) server_negotiation.cc:299] Negotiation successful
0924 09:56:25.925654 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1359,"thread_start_us":1118,"threads_started":1}
I20260924 09:56:25.932719 6595 raft_consensus.cc:348] T 00000000000000000000000000000000 P c65adc94d5fe46cb8d1244957f570265 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c65adc94d5fe46cb8d1244957f570265" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43811 } }
I20260924 09:56:25.933017 6595 raft_consensus.cc:374] T 00000000000000000000000000000000 P c65adc94d5fe46cb8d1244957f570265 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 09:56:25.933096 6595 raft_consensus.cc:729] T 00000000000000000000000000000000 P c65adc94d5fe46cb8d1244957f570265 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c65adc94d5fe46cb8d1244957f570265, State: Initialized, Role: FOLLOWER
I20260924 09:56:25.933570 6595 consensus_queue.cc:260] T 00000000000000000000000000000000 P c65adc94d5fe46cb8d1244957f570265 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c65adc94d5fe46cb8d1244957f570265" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43811 } }
I20260924 09:56:25.933751 6595 raft_consensus.cc:388] T 00000000000000000000000000000000 P c65adc94d5fe46cb8d1244957f570265 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 09:56:25.933837 6595 raft_consensus.cc:482] T 00000000000000000000000000000000 P c65adc94d5fe46cb8d1244957f570265 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 09:56:25.933969 6595 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c65adc94d5fe46cb8d1244957f570265 [term 0 FOLLOWER]: Advancing to term 1
I20260924 09:56:25.936128 6595 raft_consensus.cc:504] T 00000000000000000000000000000000 P c65adc94d5fe46cb8d1244957f570265 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c65adc94d5fe46cb8d1244957f570265" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43811 } }
I20260924 09:56:25.936563 6595 leader_election.cc:302] T 00000000000000000000000000000000 P c65adc94d5fe46cb8d1244957f570265 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c65adc94d5fe46cb8d1244957f570265; no voters:
I20260924 09:56:25.937058 6595 leader_election.cc:288] T 00000000000000000000000000000000 P c65adc94d5fe46cb8d1244957f570265 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 09:56:25.937155 6600 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c65adc94d5fe46cb8d1244957f570265 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 09:56:25.937721 6600 raft_consensus.cc:686] T 00000000000000000000000000000000 P c65adc94d5fe46cb8d1244957f570265 [term 1 LEADER]: Becoming Leader. State: Replica: c65adc94d5fe46cb8d1244957f570265, State: Running, Role: LEADER
I20260924 09:56:25.938273 6600 consensus_queue.cc:237] T 00000000000000000000000000000000 P c65adc94d5fe46cb8d1244957f570265 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c65adc94d5fe46cb8d1244957f570265" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43811 } }
I20260924 09:56:25.938999 6595 sys_catalog.cc:564] T 00000000000000000000000000000000 P c65adc94d5fe46cb8d1244957f570265 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 09:56:25.940881 6602 sys_catalog.cc:455] T 00000000000000000000000000000000 P c65adc94d5fe46cb8d1244957f570265 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c65adc94d5fe46cb8d1244957f570265. Latest consensus state: current_term: 1 leader_uuid: "c65adc94d5fe46cb8d1244957f570265" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c65adc94d5fe46cb8d1244957f570265" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43811 } } }
I20260924 09:56:25.940929 6601 sys_catalog.cc:455] T 00000000000000000000000000000000 P c65adc94d5fe46cb8d1244957f570265 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c65adc94d5fe46cb8d1244957f570265" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c65adc94d5fe46cb8d1244957f570265" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43811 } } }
I20260924 09:56:25.941233 6602 sys_catalog.cc:458] T 00000000000000000000000000000000 P c65adc94d5fe46cb8d1244957f570265 [sys.catalog]: This master's current role is: LEADER
I20260924 09:56:25.941233 6601 sys_catalog.cc:458] T 00000000000000000000000000000000 P c65adc94d5fe46cb8d1244957f570265 [sys.catalog]: This master's current role is: LEADER
I20260924 09:56:25.944711 6605 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 09:56:25.949681 6605 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 09:56:25.958381 6605 catalog_manager.cc:1347] Generated new cluster ID: 0d84da8a40b44bcdbd4e55f25ff820dc
I20260924 09:56:25.958488 6605 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 09:56:25.965158 6605 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 09:56:25.966468 6605 catalog_manager.cc:1504] Loading token signing keys...
I20260924 09:56:25.976028 6605 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c65adc94d5fe46cb8d1244957f570265: Generated new TSK 0
I20260924 09:56:25.976830 6605 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790243735634835-388-0/krb5kdc/oryx_127.0.97.1.keytab.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790243735634835-388-0/krb5kdc/oryx_127.0.97.1.keytab.
I20260924 09:56:26.022769 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
/tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790243735634835-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790243735634835-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790243735634835-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790243735634835-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43811
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790243735634835-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790243735634835-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790243735634835-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790243735634835-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 09:56:26.332365 6621 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 09:56:26.332679 6621 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 09:56:26.332772 6621 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 09:56:26.340193 6621 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 09:56:26.340313 6621 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 09:56:26.340438 6621 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260924 09:56:26.350440 6621 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790243735634835-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790243735634835-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790243735634835-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790243735634835-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43811
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790243735634835-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 09:56:26.351744 6621 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 09:56:26.353000 6621 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 09:56:26.364959 6626 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:56:26.364917 6627 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:56:26.364998 6629 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 09:56:26.365976 6621 server_base.cc:1034] running on GCE node
I20260924 09:56:26.366541 6621 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 09:56:26.367463 6621 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 09:56:26.368649 6621 hybrid_clock.cc:648] HybridClock initialized: now 1790243786368580 us; error 66 us; skew 500 ppm
I20260924 09:56:26.368938 6621 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 09:56:26 dist-test-slave-w8d2 krb5kdc[6515](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243786, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 09:56:26.371886 6621 init.cc:377] Logged in from keytab as oryx/127.0.97.1@KRBTEST.COM (short username oryx)
I20260924 09:56:26.373216 6621 webserver.cc:460] Webserver started at http://127.0.97.1:38193/ using document root <none> and password file <none>
I20260924 09:56:26.373878 6621 fs_manager.cc:359] Metadata directory not provided
I20260924 09:56:26.373988 6621 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 09:56:26.374257 6621 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 09:56:26.377094 6621 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790243735634835-388-0/minicluster-data/ts-0/data/instance:
uuid: "87e95d359d14496285f25e40421c70c8"
format_stamp: "Formatted at 2026-09-24 09:56:26 on dist-test-slave-w8d2"
server_key: "682627180f64f6ab86ad61cbf8a2fd93"
server_key_iv: "1bf7ed6f2cccf0bf3288b0eba6064106"
server_key_version: "encryptionkey@0"
I20260924 09:56:26.377830 6621 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790243735634835-388-0/minicluster-data/ts-0/wal/instance:
uuid: "87e95d359d14496285f25e40421c70c8"
format_stamp: "Formatted at 2026-09-24 09:56:26 on dist-test-slave-w8d2"
server_key: "682627180f64f6ab86ad61cbf8a2fd93"
server_key_iv: "1bf7ed6f2cccf0bf3288b0eba6064106"
server_key_version: "encryptionkey@0"
I20260924 09:56:26.382664 6621 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260924 09:56:26.386006 6636 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 09:56:26.387415 6621 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 09:56:26.387557 6621 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790243735634835-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790243735634835-388-0/minicluster-data/ts-0/wal
uuid: "87e95d359d14496285f25e40421c70c8"
format_stamp: "Formatted at 2026-09-24 09:56:26 on dist-test-slave-w8d2"
server_key: "682627180f64f6ab86ad61cbf8a2fd93"
server_key_iv: "1bf7ed6f2cccf0bf3288b0eba6064106"
server_key_version: "encryptionkey@0"
I20260924 09:56:26.387735 6621 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790243735634835-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790243735634835-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790243735634835-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 09:56:26.402667 6621 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 09:56:26.406319 6621 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 09:56:26.406563 6621 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 09:56:26.407482 6621 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 09:56:26.408728 6621 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 09:56:26.408802 6621 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 09:56:26.408923 6621 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 09:56:26.408978 6621 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 09:56:26.436995 6621 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:46191
I20260924 09:56:26.437034 6748 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:46191 every 8 connection(s)
I20260924 09:56:26.438216 6621 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790243735634835-388-0/minicluster-data/ts-0/data/info.pb
I20260924 09:56:26.440568 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu as pid 6621
I20260924 09:56:26.440758 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790243735634835-388-0/minicluster-data/ts-0/wal/instance
I20260924 09:56:26.441459 388 external_mini_cluster.cc:1428] Setting key 420c0d32254edc81ac874be1d288d7b9
Sep 24 09:56:26 dist-test-slave-w8d2 krb5kdc[6515](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243786, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260924 09:56:26.458922 6751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:26.440638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:42527 (local address 127.0.97.62:43811)
0924 09:56:26.441082 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0924 09:56:26.441089 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 09:56:26.442104 (+ 1015us) server_negotiation.cc:407] Connection header received
0924 09:56:26.444153 (+ 2049us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:56:26.444158 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:56:26.444218 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:56:26.444342 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:56:26.446144 (+ 1802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:26.446861 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:26.448928 (+ 2067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:26.449321 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:26.449420 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:26.452922 (+ 3502us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:56:26.452929 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:56:26.452933 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:56:26.452970 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:56:26.455936 (+ 2966us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:56:26.456589 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:56:26.456594 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:56:26.456598 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:56:26.456741 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:56:26.457097 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:56:26.457101 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:56:26.457104 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:56:26.457430 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:56:26.457583 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0924 09:56:26.458333 (+ 750us) server_negotiation.cc:299] Negotiation successful
0924 09:56:26.458548 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":208,"threads_started":1}
I20260924 09:56:26.459764 6752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:26.441182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43811 (local address 127.0.97.1:42527)
0924 09:56:26.441935 (+ 753us) negotiation.cc:107] Waiting for socket to connect
0924 09:56:26.441980 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 09:56:26.443867 (+ 1887us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:56:26.444567 (+ 700us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:56:26.444579 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:56:26.445257 (+ 678us) client_negotiation.cc:189] Negotiated authn=SASL
0924 09:56:26.445987 (+ 730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:26.446010 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:26.446999 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:26.447005 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:26.448772 (+ 1767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:26.448786 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:26.449533 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:26.449540 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:26.449741 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:26.450416 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 09:56:26.450436 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 09:56:26.452692 (+ 2256us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 09:56:26.456059 (+ 3367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 09:56:26.456066 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 09:56:26.456080 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 09:56:26.456476 (+ 396us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 09:56:26.456833 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 09:56:26.456836 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 09:56:26.456838 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 09:56:26.457003 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 09:56:26.457637 (+ 634us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 09:56:26.457643 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 09:56:26.457950 (+ 307us) client_negotiation.cc:769] Sending connection context
0924 09:56:26.458206 (+ 256us) client_negotiation.cc:240] Negotiation successful
0924 09:56:26.458638 (+ 432us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":299,"threads_started":1}
WARNING: no policy specified for oryx/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260924 09:56:26.461258 6749 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43811
I20260924 09:56:26.461529 6749 heartbeater.cc:464] Registering TS with master...
I20260924 09:56:26.462231 6749 heartbeater.cc:511] Master 127.0.97.62:43811 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.2@KRBTEST.COM" created.
I20260924 09:56:26.464285 6560 ts_manager.cc:194] Registered new tserver with Master: 87e95d359d14496285f25e40421c70c8 (127.0.97.1:46191)
I20260924 09:56:26.466135 6560 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:42527
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790243735634835-388-0/krb5kdc/oryx_127.0.97.2.keytab.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790243735634835-388-0/krb5kdc/oryx_127.0.97.2.keytab.
I20260924 09:56:26.486124 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
/tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790243735634835-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790243735634835-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790243735634835-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790243735634835-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43811
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790243735634835-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790243735634835-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790243735634835-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790243735634835-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 09:56:26.789419 6755 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 09:56:26.789675 6755 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 09:56:26.789798 6755 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 09:56:26.796814 6755 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 09:56:26.796926 6755 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 09:56:26.797072 6755 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260924 09:56:26.806466 6755 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790243735634835-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790243735634835-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790243735634835-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790243735634835-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43811
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790243735634835-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 09:56:26.807744 6755 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 09:56:26.808969 6755 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 09:56:26.820992 6761 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:56:26.821031 6763 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:56:26.821213 6760 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 09:56:26.821815 6755 server_base.cc:1034] running on GCE node
I20260924 09:56:26.822329 6755 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 09:56:26.823139 6755 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 09:56:26.824326 6755 hybrid_clock.cc:648] HybridClock initialized: now 1790243786824268 us; error 51 us; skew 500 ppm
I20260924 09:56:26.824625 6755 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 09:56:26 dist-test-slave-w8d2 krb5kdc[6515](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243786, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 09:56:26.827461 6755 init.cc:377] Logged in from keytab as oryx/127.0.97.2@KRBTEST.COM (short username oryx)
I20260924 09:56:26.828712 6755 webserver.cc:460] Webserver started at http://127.0.97.2:38369/ using document root <none> and password file <none>
I20260924 09:56:26.829334 6755 fs_manager.cc:359] Metadata directory not provided
I20260924 09:56:26.829429 6755 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 09:56:26.829658 6755 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 09:56:26.832403 6755 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790243735634835-388-0/minicluster-data/ts-1/data/instance:
uuid: "3dc89c3746aa49159412f76b46758c66"
format_stamp: "Formatted at 2026-09-24 09:56:26 on dist-test-slave-w8d2"
server_key: "cca81d8bfd33aa04c3e710b4c31e0242"
server_key_iv: "0d1263420cf2f239589705a7aa1e6fb8"
server_key_version: "encryptionkey@0"
I20260924 09:56:26.833114 6755 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790243735634835-388-0/minicluster-data/ts-1/wal/instance:
uuid: "3dc89c3746aa49159412f76b46758c66"
format_stamp: "Formatted at 2026-09-24 09:56:26 on dist-test-slave-w8d2"
server_key: "cca81d8bfd33aa04c3e710b4c31e0242"
server_key_iv: "0d1263420cf2f239589705a7aa1e6fb8"
server_key_version: "encryptionkey@0"
I20260924 09:56:26.837961 6755 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260924 09:56:26.841197 6770 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 09:56:26.842406 6755 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 09:56:26.842525 6755 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790243735634835-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790243735634835-388-0/minicluster-data/ts-1/wal
uuid: "3dc89c3746aa49159412f76b46758c66"
format_stamp: "Formatted at 2026-09-24 09:56:26 on dist-test-slave-w8d2"
server_key: "cca81d8bfd33aa04c3e710b4c31e0242"
server_key_iv: "0d1263420cf2f239589705a7aa1e6fb8"
server_key_version: "encryptionkey@0"
I20260924 09:56:26.842679 6755 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790243735634835-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790243735634835-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790243735634835-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 09:56:26.869794 6755 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 09:56:26.873412 6755 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 09:56:26.873633 6755 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 09:56:26.874460 6755 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 09:56:26.875770 6755 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 09:56:26.875839 6755 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 09:56:26.875924 6755 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 09:56:26.875965 6755 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 09:56:26.906329 6755 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:35061
I20260924 09:56:26.906363 6882 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:35061 every 8 connection(s)
I20260924 09:56:26.907552 6755 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790243735634835-388-0/minicluster-data/ts-1/data/info.pb
I20260924 09:56:26.911602 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu as pid 6755
I20260924 09:56:26.911772 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790243735634835-388-0/minicluster-data/ts-1/wal/instance
I20260924 09:56:26.912159 388 external_mini_cluster.cc:1428] Setting key e68237a1d719802ee9cd3a9ee9342868
Sep 24 09:56:26 dist-test-slave-w8d2 krb5kdc[6515](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243786, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260924 09:56:26.927148 6751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:26.909960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:44549 (local address 127.0.97.62:43811)
0924 09:56:26.910144 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0924 09:56:26.910150 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 09:56:26.911320 (+ 1170us) server_negotiation.cc:407] Connection header received
0924 09:56:26.913732 (+ 2412us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:56:26.913738 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:56:26.913812 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:56:26.913909 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:56:26.915570 (+ 1661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:26.916469 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:26.917902 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:26.918447 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:26.918558 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:26.922155 (+ 3597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:56:26.922159 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:56:26.922162 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:56:26.922191 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:56:26.924447 (+ 2256us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:56:26.925184 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:56:26.925188 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:56:26.925190 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:56:26.925310 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:56:26.925693 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:56:26.925696 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:56:26.925697 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:56:26.925909 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:56:26.926014 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0924 09:56:26.926764 (+ 750us) server_negotiation.cc:299] Negotiation successful
0924 09:56:26.926924 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 09:56:26.928505 6885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:26.910452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43811 (local address 127.0.97.2:44549)
0924 09:56:26.911156 (+ 704us) negotiation.cc:107] Waiting for socket to connect
0924 09:56:26.911201 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 09:56:26.913470 (+ 2269us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:56:26.914119 (+ 649us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:56:26.914130 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:56:26.914744 (+ 614us) client_negotiation.cc:189] Negotiated authn=SASL
0924 09:56:26.915421 (+ 677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:26.915439 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:26.916582 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:26.916586 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:26.917777 (+ 1191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:26.917787 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:26.918611 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:26.918616 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:26.918792 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:26.919475 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 09:56:26.919503 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0924 09:56:26.921971 (+ 2468us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 09:56:26.924578 (+ 2607us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 09:56:26.924585 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 09:56:26.924600 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 09:56:26.925092 (+ 492us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 09:56:26.925395 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 09:56:26.925399 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 09:56:26.925402 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 09:56:26.925608 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 09:56:26.926009 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 09:56:26.926017 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 09:56:26.926399 (+ 382us) client_negotiation.cc:769] Sending connection context
0924 09:56:26.926668 (+ 269us) client_negotiation.cc:240] Negotiation successful
0924 09:56:26.927158 (+ 490us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":270,"threads_started":1}
I20260924 09:56:26.929785 6883 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43811
I20260924 09:56:26.930099 6883 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for oryx/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260924 09:56:26.930907 6883 heartbeater.cc:511] Master 127.0.97.62:43811 requested a full tablet report, sending...
I20260924 09:56:26.932221 6560 ts_manager.cc:194] Registered new tserver with Master: 3dc89c3746aa49159412f76b46758c66 (127.0.97.2:35061)
I20260924 09:56:26.932948 6560 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:44549
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790243735634835-388-0/krb5kdc/oryx_127.0.97.3.keytab.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790243735634835-388-0/krb5kdc/oryx_127.0.97.3.keytab.
I20260924 09:56:26.954274 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
/tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790243735634835-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790243735634835-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790243735634835-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790243735634835-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43811
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790243735634835-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790243735634835-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790243735634835-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790243735634835-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 09:56:27.252818 6888 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 09:56:27.253132 6888 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 09:56:27.253275 6888 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 09:56:27.260458 6888 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 09:56:27.260581 6888 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 09:56:27.260731 6888 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260924 09:56:27.270467 6888 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790243735634835-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790243735634835-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790243735634835-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790243735634835-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43811
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790243735634835-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 09:56:27.271742 6888 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 09:56:27.272989 6888 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 09:56:27.285310 6894 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:56:27.285386 6896 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:56:27.285329 6893 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 09:56:27.285988 6888 server_base.cc:1034] running on GCE node
I20260924 09:56:27.286592 6888 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 09:56:27.287493 6888 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 09:56:27.288674 6888 hybrid_clock.cc:648] HybridClock initialized: now 1790243787288624 us; error 39 us; skew 500 ppm
I20260924 09:56:27.288970 6888 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 09:56:27 dist-test-slave-w8d2 krb5kdc[6515](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243787, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 09:56:27.295321 6888 init.cc:377] Logged in from keytab as oryx/127.0.97.3@KRBTEST.COM (short username oryx)
I20260924 09:56:27.296723 6888 webserver.cc:460] Webserver started at http://127.0.97.3:34619/ using document root <none> and password file <none>
I20260924 09:56:27.297398 6888 fs_manager.cc:359] Metadata directory not provided
I20260924 09:56:27.297503 6888 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 09:56:27.297770 6888 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 09:56:27.300695 6888 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790243735634835-388-0/minicluster-data/ts-2/data/instance:
uuid: "2bdc5f8429c3414e9f322f85ce09088a"
format_stamp: "Formatted at 2026-09-24 09:56:27 on dist-test-slave-w8d2"
server_key: "106fb2e9b3c52fb2c0f57effc7d9722e"
server_key_iv: "5deaea5cae45176116522466c02f07e7"
server_key_version: "encryptionkey@0"
I20260924 09:56:27.301430 6888 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790243735634835-388-0/minicluster-data/ts-2/wal/instance:
uuid: "2bdc5f8429c3414e9f322f85ce09088a"
format_stamp: "Formatted at 2026-09-24 09:56:27 on dist-test-slave-w8d2"
server_key: "106fb2e9b3c52fb2c0f57effc7d9722e"
server_key_iv: "5deaea5cae45176116522466c02f07e7"
server_key_version: "encryptionkey@0"
I20260924 09:56:27.306358 6888 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 09:56:27.309600 6903 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 09:56:27.310851 6888 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 09:56:27.310998 6888 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790243735634835-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790243735634835-388-0/minicluster-data/ts-2/wal
uuid: "2bdc5f8429c3414e9f322f85ce09088a"
format_stamp: "Formatted at 2026-09-24 09:56:27 on dist-test-slave-w8d2"
server_key: "106fb2e9b3c52fb2c0f57effc7d9722e"
server_key_iv: "5deaea5cae45176116522466c02f07e7"
server_key_version: "encryptionkey@0"
I20260924 09:56:27.311180 6888 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790243735634835-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790243735634835-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790243735634835-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 09:56:27.324721 6888 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 09:56:27.328557 6888 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 09:56:27.328804 6888 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 09:56:27.329711 6888 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 09:56:27.331039 6888 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 09:56:27.331111 6888 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 09:56:27.331218 6888 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 09:56:27.331310 6888 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260924 09:56:27.360122 6888 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:33087
I20260924 09:56:27.360167 7015 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:33087 every 8 connection(s)
I20260924 09:56:27.361310 6888 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790243735634835-388-0/minicluster-data/ts-2/data/info.pb
I20260924 09:56:27.369668 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu as pid 6888
I20260924 09:56:27.369839 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790243735634835-388-0/minicluster-data/ts-2/wal/instance
I20260924 09:56:27.370230 388 external_mini_cluster.cc:1428] Setting key 3a4598c399ef0598eadf54d5edf35804
Sep 24 09:56:27 dist-test-slave-w8d2 krb5kdc[6515](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243787, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260924 09:56:27.379468 6751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:27.363650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:53787 (local address 127.0.97.62:43811)
0924 09:56:27.363798 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0924 09:56:27.363802 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 09:56:27.364940 (+ 1138us) server_negotiation.cc:407] Connection header received
0924 09:56:27.366628 (+ 1688us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:56:27.366632 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:56:27.366698 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:56:27.366785 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:56:27.368231 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:27.368935 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:27.370445 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:27.370935 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:27.371045 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:27.374566 (+ 3521us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:56:27.374570 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:56:27.374573 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:56:27.374603 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:56:27.377001 (+ 2398us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:56:27.377621 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:56:27.377625 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:56:27.377627 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:56:27.377771 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:56:27.378095 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:56:27.378098 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:56:27.378100 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:56:27.378347 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:56:27.378453 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0924 09:56:27.379035 (+ 582us) server_negotiation.cc:299] Negotiation successful
0924 09:56:27.379182 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 09:56:27.380379 7018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:27.364110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43811 (local address 127.0.97.3:53787)
0924 09:56:27.364778 (+ 668us) negotiation.cc:107] Waiting for socket to connect
0924 09:56:27.364817 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 09:56:27.366380 (+ 1563us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:56:27.366987 (+ 607us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:56:27.366995 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:56:27.367546 (+ 551us) client_negotiation.cc:189] Negotiated authn=SASL
0924 09:56:27.368091 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:27.368108 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:27.369047 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:27.369050 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:27.370307 (+ 1257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:27.370322 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:27.371146 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:27.371152 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:27.371463 (+ 311us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:27.372112 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 09:56:27.372143 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0924 09:56:27.374380 (+ 2237us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 09:56:27.377106 (+ 2726us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 09:56:27.377111 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 09:56:27.377123 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0924 09:56:27.377497 (+ 374us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 09:56:27.377854 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 09:56:27.377857 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 09:56:27.377859 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 09:56:27.378018 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 09:56:27.378437 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 09:56:27.378442 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 09:56:27.378729 (+ 287us) client_negotiation.cc:769] Sending connection context
0924 09:56:27.378934 (+ 205us) client_negotiation.cc:240] Negotiation successful
0924 09:56:27.379325 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":274,"threads_started":1}
I20260924 09:56:27.381657 7016 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43811
I20260924 09:56:27.381948 7016 heartbeater.cc:464] Registering TS with master...
I20260924 09:56:27.382611 7016 heartbeater.cc:511] Master 127.0.97.62:43811 requested a full tablet report, sending...
I20260924 09:56:27.384218 6559 ts_manager.cc:194] Registered new tserver with Master: 2bdc5f8429c3414e9f322f85ce09088a (127.0.97.3:33087)
I20260924 09:56:27.385157 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 09:56:27.385190 6559 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:53787
WARNING: no policy specified for oryx/127.0.97.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790243735634835-388-0/krb5kdc/oryx_127.0.97.61.keytab.
Entry for principal oryx/127.0.97.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790243735634835-388-0/krb5kdc/oryx_127.0.97.61.keytab.
I20260924 09:56:27.426317 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
/tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790243735634835-388-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790243735634835-388-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790243735634835-388-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790243735634835-388-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:46619
--webserver_interface=127.0.97.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790243735634835-388-0/krb5kdc/oryx_127.0.97.61.keytab
--principal=oryx/127.0.97.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:43811,127.0.97.61:46619
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790243735634835-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790243735634835-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790243735634835-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 09:56:27.468966 6749 heartbeater.cc:503] Master 127.0.97.62:43811 was elected leader, sending a full tablet report...
W20260924 09:56:27.724218 7021 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 09:56:27.724504 7021 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 09:56:27.724633 7021 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 09:56:27.731606 7021 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 09:56:27.731709 7021 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 09:56:27.731782 7021 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 09:56:27.731858 7021 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 09:56:27.731913 7021 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 09:56:27.741401 7021 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790243735634835-388-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790243735634835-388-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:43811,127.0.97.61:46619
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790243735634835-388-0/krb5kdc/oryx_127.0.97.61.keytab
--principal=oryx/127.0.97.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:46619
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790243735634835-388-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.97.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790243735634835-388-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
W20260924 09:56:27.741739 7021 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.0.97.62:43811,127.0.97.61:46619'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260924 09:56:27.742740 7021 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 09:56:27.743983 7021 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 09:56:27.754695 7027 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:56:27.754597 7026 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:56:27.754647 7029 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 09:56:27.760448 7021 server_base.cc:1034] running on GCE node
I20260924 09:56:27.761138 7021 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 09:56:27.762420 7021 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 09:56:27.763618 7021 hybrid_clock.cc:648] HybridClock initialized: now 1790243787763556 us; error 38 us; skew 500 ppm
I20260924 09:56:27.764020 7021 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 09:56:27 dist-test-slave-w8d2 krb5kdc[6515](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243787, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 09:56:27.766973 7021 init.cc:377] Logged in from keytab as oryx/127.0.97.61@KRBTEST.COM (short username oryx)
I20260924 09:56:27.768370 7021 webserver.cc:460] Webserver started at http://127.0.97.61:36839/ using document root <none> and password file <none>
I20260924 09:56:27.769019 7021 fs_manager.cc:359] Metadata directory not provided
I20260924 09:56:27.769101 7021 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 09:56:27.769371 7021 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 09:56:27.772176 7021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790243735634835-388-0/minicluster-data/master-1/data/instance:
uuid: "b3e4c99bb58342dc915b81f8ea383475"
format_stamp: "Formatted at 2026-09-24 09:56:27 on dist-test-slave-w8d2"
server_key: "897b05488ab5229e3cc11ffec324ccc0"
server_key_iv: "60452948fe12adf256d622e340e74cf7"
server_key_version: "encryptionkey@0"
I20260924 09:56:27.772881 7021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790243735634835-388-0/minicluster-data/master-1/wal/instance:
uuid: "b3e4c99bb58342dc915b81f8ea383475"
format_stamp: "Formatted at 2026-09-24 09:56:27 on dist-test-slave-w8d2"
server_key: "897b05488ab5229e3cc11ffec324ccc0"
server_key_iv: "60452948fe12adf256d622e340e74cf7"
server_key_version: "encryptionkey@0"
I20260924 09:56:27.777729 7021 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 09:56:27.780829 7036 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 09:56:27.782083 7021 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 09:56:27.782202 7021 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790243735634835-388-0/minicluster-data/master-1/data,/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790243735634835-388-0/minicluster-data/master-1/wal
uuid: "b3e4c99bb58342dc915b81f8ea383475"
format_stamp: "Formatted at 2026-09-24 09:56:27 on dist-test-slave-w8d2"
server_key: "897b05488ab5229e3cc11ffec324ccc0"
server_key_iv: "60452948fe12adf256d622e340e74cf7"
server_key_version: "encryptionkey@0"
I20260924 09:56:27.782387 7021 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790243735634835-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790243735634835-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790243735634835-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 09:56:27.802002 7021 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 09:56:27.806666 7021 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 09:56:27.806874 7021 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 09:56:27.822950 7021 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:46619
I20260924 09:56:27.822969 7087 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:46619 every 8 connection(s)
I20260924 09:56:27.824249 7021 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790243735634835-388-0/minicluster-data/master-1/data/info.pb
I20260924 09:56:27.827742 7088 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 09:56:27.832562 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu as pid 7021
I20260924 09:56:27.834111 7088 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43811 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 09:56:27 dist-test-slave-w8d2 krb5kdc[6515](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243787, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.61@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
Sep 24 09:56:27 dist-test-slave-w8d2 krb5kdc[6515](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243785, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260924 09:56:27.851697 6751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:27.836963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52586 (local address 127.0.97.62:43811)
0924 09:56:27.837109 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0924 09:56:27.837114 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 09:56:27.837805 (+ 691us) server_negotiation.cc:407] Connection header received
0924 09:56:27.838065 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:56:27.838070 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:56:27.838143 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:56:27.838222 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:56:27.839487 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:27.840270 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:27.841684 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:27.842090 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:27.842191 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:27.845772 (+ 3581us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:56:27.845777 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:56:27.845781 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:56:27.845811 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:56:27.848451 (+ 2640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:56:27.849382 (+ 931us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:56:27.849385 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:56:27.849387 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:56:27.849494 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:56:27.849863 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:56:27.849868 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:56:27.849871 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:56:27.850260 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:56:27.850392 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0924 09:56:27.851159 (+ 767us) server_negotiation.cc:299] Negotiation successful
0924 09:56:27.851391 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 09:56:27.852860 7091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:27.834581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53154 (local address 127.0.97.61:46619)
0924 09:56:27.835357 (+ 776us) server_negotiation.cc:206] Beginning negotiation
0924 09:56:27.835380 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0924 09:56:27.835433 (+ 53us) server_negotiation.cc:407] Connection header received
0924 09:56:27.837397 (+ 1964us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:56:27.837423 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:56:27.837874 (+ 451us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:56:27.838107 (+ 233us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:56:27.839546 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:27.840712 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:27.841850 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:27.842361 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:27.842498 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:27.846471 (+ 3973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:56:27.846479 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:56:27.846491 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:56:27.846522 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:56:27.849517 (+ 2995us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:56:27.850182 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:56:27.850187 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:56:27.850190 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:56:27.850290 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:56:27.850628 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:56:27.850632 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:56:27.850635 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:56:27.850999 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:56:27.851266 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0924 09:56:27.851631 (+ 365us) server_negotiation.cc:299] Negotiation successful
0924 09:56:27.851836 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":43,"server-negotiator.queue_time_us":515,"spinlock_wait_cycles":26880,"thread_start_us":327,"threads_started":1}
I20260924 09:56:27.852877 7092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:27.837031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43811 (local address 127.0.0.1:52586)
0924 09:56:27.837654 (+ 623us) negotiation.cc:107] Waiting for socket to connect
0924 09:56:27.837695 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 09:56:27.837859 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:56:27.838332 (+ 473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:56:27.838340 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:56:27.838952 (+ 612us) client_negotiation.cc:189] Negotiated authn=SASL
0924 09:56:27.839337 (+ 385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:27.839356 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:27.840396 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:27.840401 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:27.841559 (+ 1158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:27.841570 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:27.842337 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:27.842344 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:27.842491 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:27.843257 (+ 766us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 09:56:27.843295 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0924 09:56:27.845584 (+ 2289us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 09:56:27.848575 (+ 2991us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 09:56:27.848584 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 09:56:27.848599 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 09:56:27.849276 (+ 677us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 09:56:27.849588 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 09:56:27.849593 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 09:56:27.849596 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 09:56:27.849758 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 09:56:27.850386 (+ 628us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 09:56:27.850394 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 09:56:27.850728 (+ 334us) client_negotiation.cc:769] Sending connection context
0924 09:56:27.851064 (+ 336us) client_negotiation.cc:240] Negotiation successful
0924 09:56:27.851496 (+ 432us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":438,"mutex_wait_us":4,"thread_start_us":330,"threads_started":1}
I20260924 09:56:27.854728 7088 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46619 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 09:56:27 dist-test-slave-w8d2 krb5kdc[6515](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243787, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.61@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260924 09:56:27.865675 7092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:27.855498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:46619 (local address 127.0.0.1:53170)
0924 09:56:27.855632 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0924 09:56:27.855649 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 09:56:27.855745 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:56:27.855999 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:56:27.856003 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:56:27.856308 (+ 305us) client_negotiation.cc:189] Negotiated authn=SASL
0924 09:56:27.856424 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:27.856432 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:27.857330 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:27.857335 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:27.858216 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:27.858226 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:27.858840 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:27.858845 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:27.859035 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:27.859533 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 09:56:27.859548 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 09:56:27.861279 (+ 1731us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 09:56:27.863674 (+ 2395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 09:56:27.863679 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 09:56:27.863683 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 09:56:27.864020 (+ 337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 09:56:27.864301 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 09:56:27.864306 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 09:56:27.864310 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 09:56:27.864416 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 09:56:27.864854 (+ 438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 09:56:27.864859 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 09:56:27.864981 (+ 122us) client_negotiation.cc:769] Sending connection context
0924 09:56:27.865087 (+ 106us) client_negotiation.cc:240] Negotiation successful
0924 09:56:27.865332 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260924 09:56:27.865757 7091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:27.855454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53170 (local address 127.0.97.61:46619)
0924 09:56:27.855622 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0924 09:56:27.855627 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 09:56:27.855703 (+ 76us) server_negotiation.cc:407] Connection header received
0924 09:56:27.855842 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:56:27.855846 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:56:27.855899 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:56:27.855964 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:56:27.856544 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:27.857205 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:27.858327 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:27.858722 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:27.858821 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:27.861432 (+ 2611us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:56:27.861436 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:56:27.861439 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:56:27.861472 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:56:27.863573 (+ 2101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:56:27.864111 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:56:27.864116 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:56:27.864119 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:56:27.864214 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:56:27.864506 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:56:27.864511 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:56:27.864514 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:56:27.864743 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:56:27.864859 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0924 09:56:27.865172 (+ 313us) server_negotiation.cc:299] Negotiation successful
0924 09:56:27.865476 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":112,"server-negotiator.queue_time_us":49}
I20260924 09:56:27.870124 7088 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3e4c99bb58342dc915b81f8ea383475: Bootstrap starting.
I20260924 09:56:27.873006 7088 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b3e4c99bb58342dc915b81f8ea383475: Neither blocks nor log segments found. Creating new log.
I20260924 09:56:27.873874 7088 log.cc:824] T 00000000000000000000000000000000 P b3e4c99bb58342dc915b81f8ea383475: Log is configured to *not* fsync() on all Append() calls
I20260924 09:56:27.876318 7088 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3e4c99bb58342dc915b81f8ea383475: No bootstrap required, opened a new log
I20260924 09:56:27.883035 7088 raft_consensus.cc:348] T 00000000000000000000000000000000 P b3e4c99bb58342dc915b81f8ea383475 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c65adc94d5fe46cb8d1244957f570265" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43811 } } peers { permanent_uuid: "b3e4c99bb58342dc915b81f8ea383475" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46619 } }
I20260924 09:56:27.883383 7088 raft_consensus.cc:374] T 00000000000000000000000000000000 P b3e4c99bb58342dc915b81f8ea383475 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 09:56:27.883458 7088 raft_consensus.cc:729] T 00000000000000000000000000000000 P b3e4c99bb58342dc915b81f8ea383475 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3e4c99bb58342dc915b81f8ea383475, State: Initialized, Role: FOLLOWER
I20260924 09:56:27.883975 7088 consensus_queue.cc:260] T 00000000000000000000000000000000 P b3e4c99bb58342dc915b81f8ea383475 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c65adc94d5fe46cb8d1244957f570265" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43811 } } peers { permanent_uuid: "b3e4c99bb58342dc915b81f8ea383475" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46619 } }
I20260924 09:56:27.884708 7094 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3e4c99bb58342dc915b81f8ea383475 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c65adc94d5fe46cb8d1244957f570265" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43811 } } peers { permanent_uuid: "b3e4c99bb58342dc915b81f8ea383475" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46619 } } }
I20260924 09:56:27.884997 7094 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3e4c99bb58342dc915b81f8ea383475 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 09:56:27.885718 7088 sys_catalog.cc:564] T 00000000000000000000000000000000 P b3e4c99bb58342dc915b81f8ea383475 [sys.catalog]: configured and running, proceeding with master startup.
W20260924 09:56:27.894016 7105 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b3e4c99bb58342dc915b81f8ea383475: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 09:56:27.894156 7105 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 09:56:27.895675 7021 master_runner.cc:428] Detected that this master b3e4c99bb58342dc915b81f8ea383475 is joining an existing cluster
I20260924 09:56:27.895773 7021 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.97.61:46619
I20260924 09:56:27.925225 6751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:27.903560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52598 (local address 127.0.97.62:43811)
0924 09:56:27.903734 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0924 09:56:27.903740 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 09:56:27.910668 (+ 6928us) server_negotiation.cc:407] Connection header received
0924 09:56:27.911155 (+ 487us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:56:27.911161 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:56:27.911248 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:56:27.911309 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:56:27.912611 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:27.913659 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:27.915787 (+ 2128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:27.916452 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:27.916542 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:27.919001 (+ 2459us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:56:27.919008 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:56:27.919013 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:56:27.919052 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:56:27.922265 (+ 3213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:56:27.923009 (+ 744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:56:27.923015 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:56:27.923019 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:56:27.923152 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:56:27.923614 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:56:27.923619 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:56:27.923622 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:56:27.923956 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:56:27.924066 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0924 09:56:27.924672 (+ 606us) server_negotiation.cc:299] Negotiation successful
0924 09:56:27.924886 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260924 09:56:27.926206 7111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:27.899674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43811 (local address 127.0.0.1:52598)
0924 09:56:27.910567 (+ 10893us) negotiation.cc:107] Waiting for socket to connect
0924 09:56:27.910587 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 09:56:27.911015 (+ 428us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:56:27.911479 (+ 464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:56:27.911484 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:56:27.912116 (+ 632us) client_negotiation.cc:189] Negotiated authn=SASL
0924 09:56:27.912450 (+ 334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:27.912460 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:27.913838 (+ 1378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:27.913847 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:27.915624 (+ 1777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:27.915637 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:27.916787 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:27.916793 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:27.916990 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:27.917631 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 09:56:27.917656 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0924 09:56:27.918780 (+ 1124us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 09:56:27.922409 (+ 3629us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 09:56:27.922414 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 09:56:27.922417 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 09:56:27.922860 (+ 443us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 09:56:27.923300 (+ 440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 09:56:27.923304 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 09:56:27.923308 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 09:56:27.923485 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 09:56:27.924161 (+ 676us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 09:56:27.924166 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 09:56:27.924351 (+ 185us) client_negotiation.cc:769] Sending connection context
0924 09:56:27.925484 (+ 1133us) client_negotiation.cc:240] Negotiation successful
0924 09:56:27.925769 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10746,"thread_start_us":11002,"threads_started":1}
I20260924 09:56:27.927733 7091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:27.899905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53172 (local address 127.0.97.61:46619)
0924 09:56:27.903400 (+ 3495us) server_negotiation.cc:206] Beginning negotiation
0924 09:56:27.903407 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 09:56:27.914103 (+ 10696us) server_negotiation.cc:407] Connection header received
0924 09:56:27.914369 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:56:27.914374 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:56:27.914436 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:56:27.914505 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:56:27.915769 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:27.916694 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:27.918380 (+ 1686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:27.919056 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:27.919138 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:27.921322 (+ 2184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:56:27.921328 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:56:27.921333 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:56:27.921364 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:56:27.924655 (+ 3291us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:56:27.925617 (+ 962us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:56:27.925622 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:56:27.925625 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:56:27.925767 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:56:27.926145 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:56:27.926154 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:56:27.926157 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:56:27.926523 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:56:27.926780 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0924 09:56:27.927225 (+ 445us) server_negotiation.cc:299] Negotiation successful
0924 09:56:27.927439 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3351}
I20260924 09:56:27.928794 7112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:27.900174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:46619 (local address 127.0.0.1:53172)
0924 09:56:27.914010 (+ 13836us) negotiation.cc:107] Waiting for socket to connect
0924 09:56:27.914035 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0924 09:56:27.914208 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:56:27.914656 (+ 448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:56:27.914660 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:56:27.915319 (+ 659us) client_negotiation.cc:189] Negotiated authn=SASL
0924 09:56:27.915658 (+ 339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:27.915666 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:27.916886 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:27.916890 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:27.918241 (+ 1351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:27.918257 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:27.919426 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:27.919432 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:27.919601 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:27.920189 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 09:56:27.920216 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0924 09:56:27.921150 (+ 934us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 09:56:27.924770 (+ 3620us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 09:56:27.924775 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 09:56:27.924778 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 09:56:27.925215 (+ 437us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 09:56:27.925891 (+ 676us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 09:56:27.925895 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 09:56:27.925898 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 09:56:27.926049 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 09:56:27.926838 (+ 789us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 09:56:27.926842 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 09:56:27.926989 (+ 147us) client_negotiation.cc:769] Sending connection context
0924 09:56:27.928139 (+ 1150us) client_negotiation.cc:240] Negotiation successful
0924 09:56:27.928404 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13713,"thread_start_us":13533,"threads_started":1}
WARNING: no policy specified for oryx/127.0.97.60@KRBTEST.COM; defaulting to no policy
I20260924 09:56:27.935753 6883 heartbeater.cc:503] Master 127.0.97.62:43811 was elected leader, sending a full tablet report...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.60@KRBTEST.COM" created.
Sep 24 09:56:27 dist-test-slave-w8d2 krb5kdc[6515](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243785, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260924 09:56:27.946923 7114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:27.932494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:46619 (local address 127.0.0.1:53184)
0924 09:56:27.933045 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0924 09:56:27.933075 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0924 09:56:27.933223 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:56:27.933712 (+ 489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:56:27.933717 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:56:27.934248 (+ 531us) client_negotiation.cc:189] Negotiated authn=SASL
0924 09:56:27.934439 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:27.934446 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:27.935974 (+ 1528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:27.935979 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:27.937372 (+ 1393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:27.937384 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:27.938110 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:27.938115 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:27.938285 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:27.938850 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 09:56:27.938877 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0924 09:56:27.940895 (+ 2018us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 09:56:27.944069 (+ 3174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 09:56:27.944082 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 09:56:27.944091 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0924 09:56:27.944682 (+ 591us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 09:56:27.945052 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 09:56:27.945057 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 09:56:27.945061 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 09:56:27.945213 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 09:56:27.945814 (+ 601us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 09:56:27.945822 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 09:56:27.945999 (+ 177us) client_negotiation.cc:769] Sending connection context
0924 09:56:27.946170 (+ 171us) client_negotiation.cc:240] Negotiation successful
0924 09:56:27.946532 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":241,"threads_started":1}
I20260924 09:56:27.946920 7091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:27.932401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53184 (local address 127.0.97.61:46619)
0924 09:56:27.932530 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0924 09:56:27.932534 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 09:56:27.933156 (+ 622us) server_negotiation.cc:407] Connection header received
0924 09:56:27.933382 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:56:27.933386 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:56:27.933460 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:56:27.933540 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:56:27.934700 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:27.935837 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:27.937505 (+ 1668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:27.937961 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:27.938066 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:27.941075 (+ 3009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:56:27.941081 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:56:27.941084 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:56:27.941111 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:56:27.943918 (+ 2807us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:56:27.944807 (+ 889us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:56:27.944812 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:56:27.944816 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:56:27.944940 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:56:27.945346 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:56:27.945350 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:56:27.945352 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:56:27.945706 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:56:27.945876 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0924 09:56:27.946291 (+ 415us) server_negotiation.cc:299] Negotiation successful
0924 09:56:27.946517 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260924 09:56:27.948961 6559 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.97.61:46619: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "b3e4c99bb58342dc915b81f8ea383475"
member_type: NON_VOTER
last_known_addr {
host: "127.0.97.61"
port: 46619
}
attrs {
promote: true
}
}
dest_uuid: "c65adc94d5fe46cb8d1244957f570265"
cas_config_opid_index: -1
I20260924 09:56:27.950042 6559 consensus_queue.cc:237] T 00000000000000000000000000000000 P c65adc94d5fe46cb8d1244957f570265 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c65adc94d5fe46cb8d1244957f570265" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43811 } } peers { permanent_uuid: "b3e4c99bb58342dc915b81f8ea383475" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 46619 } attrs { promote: true } }
I20260924 09:56:27.957500 7063 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b3e4c99bb58342dc915b81f8ea383475 [term 0 FOLLOWER]: Advancing to term 1
I20260924 09:56:27.959717 7118 raft_consensus.cc:2932] T 00000000000000000000000000000000 P c65adc94d5fe46cb8d1244957f570265 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER b3e4c99bb58342dc915b81f8ea383475 (127.0.97.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c65adc94d5fe46cb8d1244957f570265" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43811 } } peers { permanent_uuid: "b3e4c99bb58342dc915b81f8ea383475" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 46619 } attrs { promote: true } } }
I20260924 09:56:27.962678 7063 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b3e4c99bb58342dc915b81f8ea383475 [term 1 FOLLOWER]: Refusing update from remote peer c65adc94d5fe46cb8d1244957f570265: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260924 09:56:27.964352 7117 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c65adc94d5fe46cb8d1244957f570265 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3e4c99bb58342dc915b81f8ea383475" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 46619 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 09:56:27.967007 7118 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.97.61:46619
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790243735634835-388-0/krb5kdc/oryx_127.0.97.60.keytab.
Entry for principal oryx/127.0.97.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790243735634835-388-0/krb5kdc/oryx_127.0.97.60.keytab.
I20260924 09:56:27.967686 7021 master.cc:537] Master@127.0.97.61:46619 shutting down...
I20260924 09:56:27.968936 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
/tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790243735634835-388-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790243735634835-388-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790243735634835-388-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790243735634835-388-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:34989
--webserver_interface=127.0.97.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790243735634835-388-0/krb5kdc/oryx_127.0.97.60.keytab
--principal=oryx/127.0.97.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:43811,127.0.97.61:46619,127.0.97.60:34989
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790243735634835-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790243735634835-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790243735634835-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 09:56:27.968928 7117 sys_catalog.cc:455] T 00000000000000000000000000000000 P c65adc94d5fe46cb8d1244957f570265 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "c65adc94d5fe46cb8d1244957f570265" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c65adc94d5fe46cb8d1244957f570265" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43811 } } peers { permanent_uuid: "b3e4c99bb58342dc915b81f8ea383475" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 46619 } attrs { promote: true } } }
I20260924 09:56:27.969173 7117 sys_catalog.cc:458] T 00000000000000000000000000000000 P c65adc94d5fe46cb8d1244957f570265 [sys.catalog]: This master's current role is: LEADER
I20260924 09:56:28.002399 7063 raft_consensus.cc:2932] T 00000000000000000000000000000000 P b3e4c99bb58342dc915b81f8ea383475 [term 1 LEARNER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER b3e4c99bb58342dc915b81f8ea383475 (127.0.97.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c65adc94d5fe46cb8d1244957f570265" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43811 } } peers { permanent_uuid: "b3e4c99bb58342dc915b81f8ea383475" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 46619 } attrs { promote: true } } }
I20260924 09:56:28.006127 7127 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3e4c99bb58342dc915b81f8ea383475 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c65adc94d5fe46cb8d1244957f570265" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c65adc94d5fe46cb8d1244957f570265" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43811 } } peers { permanent_uuid: "b3e4c99bb58342dc915b81f8ea383475" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 46619 } attrs { promote: true } } }
I20260924 09:56:28.006637 7127 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3e4c99bb58342dc915b81f8ea383475 [sys.catalog]: This master's current role is: LEARNER
I20260924 09:56:28.007057 7124 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3e4c99bb58342dc915b81f8ea383475 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c65adc94d5fe46cb8d1244957f570265" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c65adc94d5fe46cb8d1244957f570265" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43811 } } peers { permanent_uuid: "b3e4c99bb58342dc915b81f8ea383475" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 46619 } attrs { promote: true } } }
I20260924 09:56:28.007460 7124 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3e4c99bb58342dc915b81f8ea383475 [sys.catalog]: This master's current role is: LEARNER
I20260924 09:56:28.007030 7094 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3e4c99bb58342dc915b81f8ea383475 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c65adc94d5fe46cb8d1244957f570265. Latest consensus state: current_term: 1 leader_uuid: "c65adc94d5fe46cb8d1244957f570265" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c65adc94d5fe46cb8d1244957f570265" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43811 } } peers { permanent_uuid: "b3e4c99bb58342dc915b81f8ea383475" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 46619 } attrs { promote: true } } }
I20260924 09:56:28.008144 7094 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3e4c99bb58342dc915b81f8ea383475 [sys.catalog]: This master's current role is: LEARNER
I20260924 09:56:28.025064 7021 raft_consensus.cc:2229] T 00000000000000000000000000000000 P b3e4c99bb58342dc915b81f8ea383475 [term 1 LEARNER]: Raft consensus shutting down.
I20260924 09:56:28.025547 7021 raft_consensus.cc:2258] T 00000000000000000000000000000000 P b3e4c99bb58342dc915b81f8ea383475 [term 1 LEARNER]: Raft consensus is shut down!
I20260924 09:56:28.025702 7021 tablet_replica.cc:330] T 00000000000000000000000000000000 P b3e4c99bb58342dc915b81f8ea383475: stopping tablet replica
I20260924 09:56:28.032361 7021 master.cc:552] Master@127.0.97.61:46619 shutdown complete.
I20260924 09:56:28.032449 7021 master_runner.cc:305] Clearing existing system tablet
I20260924 09:56:28.032835 7021 fs_manager.cc:359] Metadata directory not provided
I20260924 09:56:28.032919 7021 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 09:56:28.035028 7021 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 09:56:28.036490 7129 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 09:56:28.036773 7021 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790243735634835-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 09:56:28.037056 7021 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 09:56:28.037155 7021 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790243735634835-388-0/minicluster-data/master-1/data,/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790243735634835-388-0/minicluster-data/master-1/wal
uuid: "b3e4c99bb58342dc915b81f8ea383475"
format_stamp: "Formatted at 2026-09-24 09:56:27 on dist-test-slave-w8d2"
server_key: "897b05488ab5229e3cc11ffec324ccc0"
server_key_iv: "60452948fe12adf256d622e340e74cf7"
server_key_version: "encryptionkey@0"
I20260924 09:56:28.040140 7021 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P b3e4c99bb58342dc915b81f8ea383475: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 09:56:28.044687 7021 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P b3e4c99bb58342dc915b81f8ea383475: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260924 09:56:28.044931 7021 log.cc:1196] T 00000000000000000000000000000000 P b3e4c99bb58342dc915b81f8ea383475: Deleting WAL directory at /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790243735634835-388-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260924 09:56:28.045364 7021 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P b3e4c99bb58342dc915b81f8ea383475: Deleting consensus metadata
I20260924 09:56:28.045598 7021 master_runner.cc:315] Copying system tablet from 127.0.97.62:43811
I20260924 09:56:28.052284 7021 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P b3e4c99bb58342dc915b81f8ea383475: tablet copy: Beginning tablet copy session from remote peer at address 127.0.97.62:43811
I20260924 09:56:28.069985 6751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:28.052972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52602 (local address 127.0.97.62:43811)
0924 09:56:28.053162 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0924 09:56:28.053169 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 09:56:28.059837 (+ 6668us) server_negotiation.cc:407] Connection header received
0924 09:56:28.060019 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:56:28.060025 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:56:28.060103 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:56:28.060185 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:56:28.060997 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:28.061667 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:28.062864 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:28.063322 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:28.063424 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:28.065143 (+ 1719us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:56:28.065147 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:56:28.065150 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:56:28.065178 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:56:28.067637 (+ 2459us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:56:28.068193 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:56:28.068199 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:56:28.068202 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:56:28.068312 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:56:28.068652 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:56:28.068657 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:56:28.068660 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:56:28.068921 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:56:28.069052 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0924 09:56:28.069418 (+ 366us) server_negotiation.cc:299] Negotiation successful
0924 09:56:28.069630 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 09:56:28.070050 7135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:28.052951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43811 (local address 127.0.0.1:52602)
0924 09:56:28.059714 (+ 6763us) negotiation.cc:107] Waiting for socket to connect
0924 09:56:28.059732 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 09:56:28.059895 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:56:28.060242 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:56:28.060246 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:56:28.060630 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0924 09:56:28.060833 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:28.060842 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:28.061797 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:28.061803 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:28.062736 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:28.062748 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:28.063477 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:28.063482 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:28.063678 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:28.064295 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 09:56:28.064313 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 09:56:28.064964 (+ 651us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 09:56:28.067750 (+ 2786us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 09:56:28.067756 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 09:56:28.067760 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 09:56:28.068092 (+ 332us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 09:56:28.068408 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 09:56:28.068413 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 09:56:28.068420 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0924 09:56:28.068560 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 09:56:28.069033 (+ 473us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 09:56:28.069037 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 09:56:28.069185 (+ 148us) client_negotiation.cc:769] Sending connection context
0924 09:56:28.069315 (+ 130us) client_negotiation.cc:240] Negotiation successful
0924 09:56:28.069632 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6610,"thread_start_us":6474,"threads_started":1}
I20260924 09:56:28.070994 6580 tablet_copy_service.cc:140] P c65adc94d5fe46cb8d1244957f570265: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer b3e4c99bb58342dc915b81f8ea383475 ({username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:52602)
I20260924 09:56:28.071219 6580 tablet_copy_service.cc:161] P c65adc94d5fe46cb8d1244957f570265: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer b3e4c99bb58342dc915b81f8ea383475 at {username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:52602: session id = b3e4c99bb58342dc915b81f8ea383475-00000000000000000000000000000000
I20260924 09:56:28.074055 6580 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P c65adc94d5fe46cb8d1244957f570265: Tablet Copy: opened 0 blocks and 1 log segments
I20260924 09:56:28.075838 7021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260924 09:56:28.080246 7021 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P b3e4c99bb58342dc915b81f8ea383475: tablet copy: Starting download of 0 data blocks...
I20260924 09:56:28.080513 7021 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P b3e4c99bb58342dc915b81f8ea383475: tablet copy: Starting download of 1 WAL segments...
I20260924 09:56:28.082625 7021 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P b3e4c99bb58342dc915b81f8ea383475: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260924 09:56:28.085525 6580 tablet_copy_service.cc:342] P c65adc94d5fe46cb8d1244957f570265: Request end of tablet copy session b3e4c99bb58342dc915b81f8ea383475-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:52602
I20260924 09:56:28.085680 6580 tablet_copy_service.cc:433] P c65adc94d5fe46cb8d1244957f570265: ending tablet copy session b3e4c99bb58342dc915b81f8ea383475-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer b3e4c99bb58342dc915b81f8ea383475
I20260924 09:56:28.094723 7021 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 09:56:28.097107 7136 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:56:28.107575 7137 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:56:28.111539 7139 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 09:56:28.113616 7021 server_base.cc:1034] running on GCE node
I20260924 09:56:28.114006 7021 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 09:56:28.116044 7021 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 09:56:28.117295 7021 hybrid_clock.cc:648] HybridClock initialized: now 1790243788117108 us; error 245 us; skew 500 ppm
I20260924 09:56:28.117623 7021 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 09:56:28 dist-test-slave-w8d2 krb5kdc[6515](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243788, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 09:56:28.119536 7021 init.cc:377] Logged in from keytab as oryx/127.0.97.61@KRBTEST.COM (short username oryx)
I20260924 09:56:28.120683 7021 webserver.cc:460] Webserver started at http://127.0.97.61:33215/ using document root <none> and password file <none>
I20260924 09:56:28.120985 7021 fs_manager.cc:359] Metadata directory not provided
I20260924 09:56:28.121054 7021 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 09:56:28.123171 7021 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 09:56:28.124730 7146 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 09:56:28.125090 7021 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 09:56:28.125195 7021 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790243735634835-388-0/minicluster-data/master-1/data,/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790243735634835-388-0/minicluster-data/master-1/wal
uuid: "b3e4c99bb58342dc915b81f8ea383475"
format_stamp: "Formatted at 2026-09-24 09:56:27 on dist-test-slave-w8d2"
server_key: "897b05488ab5229e3cc11ffec324ccc0"
server_key_iv: "60452948fe12adf256d622e340e74cf7"
server_key_version: "encryptionkey@0"
I20260924 09:56:28.125334 7021 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790243735634835-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790243735634835-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790243735634835-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 09:56:28.150338 7021 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 09:56:28.153795 7021 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 09:56:28.179153 7021 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:46619
I20260924 09:56:28.179174 7197 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:46619 every 8 connection(s)
I20260924 09:56:28.180014 7021 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790243735634835-388-0/minicluster-data/master-1/data/info.pb
I20260924 09:56:28.185428 7198 sys_catalog.cc:263] Verifying existing consensus state
I20260924 09:56:28.189790 7198 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3e4c99bb58342dc915b81f8ea383475: Bootstrap starting.
I20260924 09:56:28.200320 7198 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3e4c99bb58342dc915b81f8ea383475: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 09:56:28.200716 7198 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3e4c99bb58342dc915b81f8ea383475: Bootstrap complete.
I20260924 09:56:28.201483 7198 raft_consensus.cc:348] T 00000000000000000000000000000000 P b3e4c99bb58342dc915b81f8ea383475 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c65adc94d5fe46cb8d1244957f570265" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43811 } } peers { permanent_uuid: "b3e4c99bb58342dc915b81f8ea383475" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 46619 } attrs { promote: true } }
I20260924 09:56:28.201673 7198 raft_consensus.cc:729] T 00000000000000000000000000000000 P b3e4c99bb58342dc915b81f8ea383475 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b3e4c99bb58342dc915b81f8ea383475, State: Initialized, Role: LEARNER
I20260924 09:56:28.201900 7198 consensus_queue.cc:260] T 00000000000000000000000000000000 P b3e4c99bb58342dc915b81f8ea383475 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c65adc94d5fe46cb8d1244957f570265" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43811 } } peers { permanent_uuid: "b3e4c99bb58342dc915b81f8ea383475" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 46619 } attrs { promote: true } }
I20260924 09:56:28.202589 7201 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3e4c99bb58342dc915b81f8ea383475 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c65adc94d5fe46cb8d1244957f570265" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43811 } } peers { permanent_uuid: "b3e4c99bb58342dc915b81f8ea383475" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 46619 } attrs { promote: true } } }
I20260924 09:56:28.202885 7198 sys_catalog.cc:564] T 00000000000000000000000000000000 P b3e4c99bb58342dc915b81f8ea383475 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 09:56:28.203001 7201 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3e4c99bb58342dc915b81f8ea383475 [sys.catalog]: This master's current role is: LEARNER
I20260924 09:56:28.213449 7212 catalog_manager.cc:1259] Loaded cluster ID: 0d84da8a40b44bcdbd4e55f25ff820dc
I20260924 09:56:28.213541 7212 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b3e4c99bb58342dc915b81f8ea383475: loading cluster ID for follower catalog manager: success
I20260924 09:56:28.216049 7212 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b3e4c99bb58342dc915b81f8ea383475: acquiring CA information for follower catalog manager: success
I20260924 09:56:28.218029 7212 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b3e4c99bb58342dc915b81f8ea383475: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 09:56:28.385727 7118 sys_catalog.cc:455] T 00000000000000000000000000000000 P c65adc94d5fe46cb8d1244957f570265 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c65adc94d5fe46cb8d1244957f570265" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c65adc94d5fe46cb8d1244957f570265" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43811 } } peers { permanent_uuid: "b3e4c99bb58342dc915b81f8ea383475" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 46619 } attrs { promote: true } } }
I20260924 09:56:28.386037 7118 sys_catalog.cc:458] T 00000000000000000000000000000000 P c65adc94d5fe46cb8d1244957f570265 [sys.catalog]: This master's current role is: LEADER
I20260924 09:56:28.387641 7016 heartbeater.cc:503] Master 127.0.97.62:43811 was elected leader, sending a full tablet report...
I20260924 09:56:28.393118 7114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:28.386049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:46619 (local address 127.0.0.1:53200)
0924 09:56:28.386344 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0924 09:56:28.386363 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 09:56:28.386496 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:56:28.387532 (+ 1036us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:56:28.387537 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:56:28.387565 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 09:56:28.387768 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:28.387780 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:28.389707 (+ 1927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:28.389712 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:28.391054 (+ 1342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:28.391070 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:28.392348 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:28.392354 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:28.392559 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:28.392622 (+ 63us) client_negotiation.cc:769] Sending connection context
0924 09:56:28.392703 (+ 81us) client_negotiation.cc:240] Negotiation successful
0924 09:56:28.392842 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164}
I20260924 09:56:28.393214 7213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:28.386544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53200 (local address 127.0.97.61:46619)
0924 09:56:28.387145 (+ 601us) server_negotiation.cc:206] Beginning negotiation
0924 09:56:28.387149 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 09:56:28.387165 (+ 16us) server_negotiation.cc:407] Connection header received
0924 09:56:28.387222 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:56:28.387226 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:56:28.387332 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:56:28.387915 (+ 583us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 09:56:28.388140 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:28.389573 (+ 1433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:28.391206 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:28.392207 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:28.392315 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:28.392557 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0924 09:56:28.392745 (+ 188us) server_negotiation.cc:299] Negotiation successful
0924 09:56:28.392875 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":482,"thread_start_us":321,"threads_started":1}
I20260924 09:56:28.394256 7201 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3e4c99bb58342dc915b81f8ea383475 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c65adc94d5fe46cb8d1244957f570265. Latest consensus state: current_term: 1 leader_uuid: "c65adc94d5fe46cb8d1244957f570265" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c65adc94d5fe46cb8d1244957f570265" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43811 } } peers { permanent_uuid: "b3e4c99bb58342dc915b81f8ea383475" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 46619 } attrs { promote: true } } }
I20260924 09:56:28.394546 7201 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3e4c99bb58342dc915b81f8ea383475 [sys.catalog]: This master's current role is: LEARNER
I20260924 09:56:28.395362 7119 raft_consensus.cc:1050] T 00000000000000000000000000000000 P c65adc94d5fe46cb8d1244957f570265: attempting to promote NON_VOTER b3e4c99bb58342dc915b81f8ea383475 to VOTER
I20260924 09:56:28.396103 7119 consensus_queue.cc:237] T 00000000000000000000000000000000 P c65adc94d5fe46cb8d1244957f570265 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c65adc94d5fe46cb8d1244957f570265" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43811 } } peers { permanent_uuid: "b3e4c99bb58342dc915b81f8ea383475" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46619 } attrs { promote: false } }
I20260924 09:56:28.397845 7173 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b3e4c99bb58342dc915b81f8ea383475 [term 1 LEARNER]: Refusing update from remote peer c65adc94d5fe46cb8d1244957f570265: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260924 09:56:28.398360 7118 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c65adc94d5fe46cb8d1244957f570265 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3e4c99bb58342dc915b81f8ea383475" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46619 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260924 09:56:28.401141 7119 raft_consensus.cc:2932] T 00000000000000000000000000000000 P c65adc94d5fe46cb8d1244957f570265 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, b3e4c99bb58342dc915b81f8ea383475 (127.0.97.61) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c65adc94d5fe46cb8d1244957f570265" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43811 } } peers { permanent_uuid: "b3e4c99bb58342dc915b81f8ea383475" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46619 } attrs { promote: false } } }
I20260924 09:56:28.403784 7119 sys_catalog.cc:455] T 00000000000000000000000000000000 P c65adc94d5fe46cb8d1244957f570265 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "c65adc94d5fe46cb8d1244957f570265" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c65adc94d5fe46cb8d1244957f570265" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43811 } } peers { permanent_uuid: "b3e4c99bb58342dc915b81f8ea383475" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46619 } attrs { promote: false } } }
I20260924 09:56:28.403998 7119 sys_catalog.cc:458] T 00000000000000000000000000000000 P c65adc94d5fe46cb8d1244957f570265 [sys.catalog]: This master's current role is: LEADER
I20260924 09:56:28.404251 7118 sys_catalog.cc:455] T 00000000000000000000000000000000 P c65adc94d5fe46cb8d1244957f570265 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c65adc94d5fe46cb8d1244957f570265" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c65adc94d5fe46cb8d1244957f570265" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43811 } } peers { permanent_uuid: "b3e4c99bb58342dc915b81f8ea383475" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46619 } attrs { promote: false } } }
I20260924 09:56:28.404440 7118 sys_catalog.cc:458] T 00000000000000000000000000000000 P c65adc94d5fe46cb8d1244957f570265 [sys.catalog]: This master's current role is: LEADER
I20260924 09:56:28.404896 7173 raft_consensus.cc:2932] T 00000000000000000000000000000000 P b3e4c99bb58342dc915b81f8ea383475 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, b3e4c99bb58342dc915b81f8ea383475 (127.0.97.61) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c65adc94d5fe46cb8d1244957f570265" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43811 } } peers { permanent_uuid: "b3e4c99bb58342dc915b81f8ea383475" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46619 } attrs { promote: false } } }
I20260924 09:56:28.408643 7201 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3e4c99bb58342dc915b81f8ea383475 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c65adc94d5fe46cb8d1244957f570265" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c65adc94d5fe46cb8d1244957f570265" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43811 } } peers { permanent_uuid: "b3e4c99bb58342dc915b81f8ea383475" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46619 } attrs { promote: false } } }
I20260924 09:56:28.408871 7201 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3e4c99bb58342dc915b81f8ea383475 [sys.catalog]: This master's current role is: FOLLOWER
W20260924 09:56:28.419493 7122 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 09:56:28.419775 7122 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 09:56:28.419911 7122 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 09:56:28.427778 7122 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 09:56:28.427896 7122 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 09:56:28.427963 7122 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 09:56:28.428035 7122 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 09:56:28.428089 7122 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 09:56:28.438109 7122 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790243735634835-388-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790243735634835-388-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:43811,127.0.97.61:46619,127.0.97.60:34989
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790243735634835-388-0/krb5kdc/oryx_127.0.97.60.keytab
--principal=oryx/127.0.97.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:34989
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790243735634835-388-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.97.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790243735634835-388-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 09:56:28.439497 7122 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 09:56:28.440852 7122 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 09:56:28.452137 7219 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:56:28.452329 7222 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:56:28.452543 7220 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 09:56:28.452991 7122 server_base.cc:1034] running on GCE node
I20260924 09:56:28.454214 7122 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 09:56:28.455732 7122 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 09:56:28.456954 7122 hybrid_clock.cc:648] HybridClock initialized: now 1790243788456901 us; error 47 us; skew 500 ppm
I20260924 09:56:28.457276 7122 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 09:56:28 dist-test-slave-w8d2 krb5kdc[6515](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243788, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 09:56:28.460258 7122 init.cc:377] Logged in from keytab as oryx/127.0.97.60@KRBTEST.COM (short username oryx)
I20260924 09:56:28.461647 7122 webserver.cc:460] Webserver started at http://127.0.97.60:35185/ using document root <none> and password file <none>
I20260924 09:56:28.462267 7122 fs_manager.cc:359] Metadata directory not provided
I20260924 09:56:28.462363 7122 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 09:56:28.462605 7122 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 09:56:28.465415 7122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790243735634835-388-0/minicluster-data/master-2/data/instance:
uuid: "3c8287837bd048888df868ab81b87c63"
format_stamp: "Formatted at 2026-09-24 09:56:28 on dist-test-slave-w8d2"
server_key: "2070cee9a1958a60ab4952e16b2765bb"
server_key_iv: "9dff8533c17ad75ba5c0360e8165b40f"
server_key_version: "encryptionkey@0"
I20260924 09:56:28.466107 7122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790243735634835-388-0/minicluster-data/master-2/wal/instance:
uuid: "3c8287837bd048888df868ab81b87c63"
format_stamp: "Formatted at 2026-09-24 09:56:28 on dist-test-slave-w8d2"
server_key: "2070cee9a1958a60ab4952e16b2765bb"
server_key_iv: "9dff8533c17ad75ba5c0360e8165b40f"
server_key_version: "encryptionkey@0"
I20260924 09:56:28.471369 7122 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260924 09:56:28.474887 7229 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 09:56:28.476282 7122 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 09:56:28.476454 7122 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790243735634835-388-0/minicluster-data/master-2/data,/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790243735634835-388-0/minicluster-data/master-2/wal
uuid: "3c8287837bd048888df868ab81b87c63"
format_stamp: "Formatted at 2026-09-24 09:56:28 on dist-test-slave-w8d2"
server_key: "2070cee9a1958a60ab4952e16b2765bb"
server_key_iv: "9dff8533c17ad75ba5c0360e8165b40f"
server_key_version: "encryptionkey@0"
I20260924 09:56:28.476615 7122 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790243735634835-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790243735634835-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790243735634835-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 09:56:28.500357 7122 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 09:56:28.504681 7122 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 09:56:28.504940 7122 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 09:56:28.521889 7122 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:34989
I20260924 09:56:28.521908 7280 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:34989 every 8 connection(s)
I20260924 09:56:28.523277 7122 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790243735634835-388-0/minicluster-data/master-2/data/info.pb
I20260924 09:56:28.526876 7281 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 09:56:28.532864 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu as pid 7122
I20260924 09:56:28.532956 7281 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43811 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 09:56:28 dist-test-slave-w8d2 krb5kdc[6515](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243788, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
Sep 24 09:56:28 dist-test-slave-w8d2 krb5kdc[6515](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243785, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
I20260924 09:56:28.552006 6751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:28.535689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52614 (local address 127.0.97.62:43811)
0924 09:56:28.535826 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0924 09:56:28.535830 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 09:56:28.536968 (+ 1138us) server_negotiation.cc:407] Connection header received
0924 09:56:28.538095 (+ 1127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:56:28.538101 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:56:28.538170 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:56:28.538250 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:56:28.539542 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:28.540303 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:28.541740 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:28.542168 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:28.542274 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:28.545695 (+ 3421us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:56:28.545702 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:56:28.545707 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:56:28.545746 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:56:28.548786 (+ 3040us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:56:28.549577 (+ 791us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:56:28.549582 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:56:28.549585 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:56:28.549720 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:56:28.550116 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:56:28.550121 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:56:28.550125 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:56:28.550468 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:56:28.550615 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0924 09:56:28.551486 (+ 871us) server_negotiation.cc:299] Negotiation successful
0924 09:56:28.551680 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 09:56:28.552942 7285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:28.535644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43811 (local address 127.0.0.1:52614)
0924 09:56:28.536830 (+ 1186us) negotiation.cc:107] Waiting for socket to connect
0924 09:56:28.536861 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0924 09:56:28.537871 (+ 1010us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:56:28.538328 (+ 457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:56:28.538335 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:56:28.538949 (+ 614us) client_negotiation.cc:189] Negotiated authn=SASL
0924 09:56:28.539376 (+ 427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:28.539394 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:28.540437 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:28.540441 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:28.541609 (+ 1168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:28.541619 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:28.542382 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:28.542390 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:28.542559 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:28.543195 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 09:56:28.543229 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0924 09:56:28.545493 (+ 2264us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 09:56:28.548909 (+ 3416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 09:56:28.548921 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 09:56:28.548934 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 09:56:28.549469 (+ 535us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 09:56:28.549845 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 09:56:28.549848 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 09:56:28.549850 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 09:56:28.550014 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 09:56:28.550596 (+ 582us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 09:56:28.550604 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 09:56:28.551044 (+ 440us) client_negotiation.cc:769] Sending connection context
0924 09:56:28.551368 (+ 324us) client_negotiation.cc:240] Negotiation successful
0924 09:56:28.551818 (+ 450us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":995,"thread_start_us":869,"threads_started":1}
I20260924 09:56:28.553458 7284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:28.534779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48316 (local address 127.0.97.60:34989)
0924 09:56:28.535580 (+ 801us) server_negotiation.cc:206] Beginning negotiation
0924 09:56:28.535593 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 09:56:28.535636 (+ 43us) server_negotiation.cc:407] Connection header received
0924 09:56:28.537864 (+ 2228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:56:28.537885 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:56:28.538116 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:56:28.538294 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:56:28.539723 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:28.540906 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:28.542052 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:28.542556 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:28.542661 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:28.547356 (+ 4695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:56:28.547367 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:56:28.547385 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:56:28.547425 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:56:28.550540 (+ 3115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:56:28.551258 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:56:28.551267 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:56:28.551271 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:56:28.551465 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:56:28.551847 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:56:28.551852 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:56:28.551856 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:56:28.552257 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:56:28.552529 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0924 09:56:28.552838 (+ 309us) server_negotiation.cc:299] Negotiation successful
0924 09:56:28.553062 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":545,"thread_start_us":302,"threads_started":1}
I20260924 09:56:28.554414 7281 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46619 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 09:56:28 dist-test-slave-w8d2 krb5kdc[6515](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243788, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260924 09:56:28.566766 7285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:28.555147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:46619 (local address 127.0.0.1:53204)
0924 09:56:28.555344 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0924 09:56:28.555362 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 09:56:28.555446 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:56:28.556010 (+ 564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:56:28.556015 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:56:28.556330 (+ 315us) client_negotiation.cc:189] Negotiated authn=SASL
0924 09:56:28.556451 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:28.556459 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:28.557378 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:28.557383 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:28.558349 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:28.558360 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:28.559025 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:28.559031 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:28.559200 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:28.559712 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 09:56:28.559728 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 09:56:28.561534 (+ 1806us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 09:56:28.564204 (+ 2670us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 09:56:28.564209 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 09:56:28.564214 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 09:56:28.564577 (+ 363us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 09:56:28.564896 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 09:56:28.564901 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 09:56:28.564904 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 09:56:28.565444 (+ 540us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 09:56:28.565906 (+ 462us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 09:56:28.565911 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 09:56:28.566037 (+ 126us) client_negotiation.cc:769] Sending connection context
0924 09:56:28.566147 (+ 110us) client_negotiation.cc:240] Negotiation successful
0924 09:56:28.566441 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260924 09:56:28.566772 7213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:28.555157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53204 (local address 127.0.97.61:46619)
0924 09:56:28.555748 (+ 591us) server_negotiation.cc:206] Beginning negotiation
0924 09:56:28.555754 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 09:56:28.555770 (+ 16us) server_negotiation.cc:407] Connection header received
0924 09:56:28.555822 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:56:28.555827 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:56:28.555896 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:56:28.555975 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:56:28.556587 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:28.557269 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:28.558470 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:28.558891 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:28.559008 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:28.561722 (+ 2714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:56:28.561727 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:56:28.561730 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:56:28.561761 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:56:28.564086 (+ 2325us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:56:28.564679 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:56:28.564684 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:56:28.564687 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:56:28.564800 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:56:28.565545 (+ 745us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:56:28.565550 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:56:28.565553 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:56:28.565802 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:56:28.565932 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0924 09:56:28.566241 (+ 309us) server_negotiation.cc:299] Negotiation successful
0924 09:56:28.566466 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":482}
I20260924 09:56:28.567782 7281 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34989 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 09:56:28 dist-test-slave-w8d2 krb5kdc[6515](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243788, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
I20260924 09:56:28.578517 7285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:28.568388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34989 (local address 127.0.0.1:48324)
0924 09:56:28.568581 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0924 09:56:28.568598 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 09:56:28.568687 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:56:28.568963 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:56:28.568968 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:56:28.569272 (+ 304us) client_negotiation.cc:189] Negotiated authn=SASL
0924 09:56:28.569398 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:28.569405 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:28.570260 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:28.570265 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:28.571066 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:28.571075 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:28.571773 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:28.571779 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:28.571956 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:28.572435 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 09:56:28.572451 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 09:56:28.574245 (+ 1794us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 09:56:28.576713 (+ 2468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 09:56:28.576717 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 09:56:28.576719 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 09:56:28.577038 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 09:56:28.577295 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 09:56:28.577298 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 09:56:28.577300 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 09:56:28.577452 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 09:56:28.577856 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 09:56:28.577859 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 09:56:28.577965 (+ 106us) client_negotiation.cc:769] Sending connection context
0924 09:56:28.578062 (+ 97us) client_negotiation.cc:240] Negotiation successful
0924 09:56:28.578235 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260924 09:56:28.578553 7284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:28.568557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48324 (local address 127.0.97.60:34989)
0924 09:56:28.568690 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0924 09:56:28.568695 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 09:56:28.568711 (+ 16us) server_negotiation.cc:407] Connection header received
0924 09:56:28.568795 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:56:28.568800 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:56:28.568850 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:56:28.568920 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:56:28.569516 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:28.570147 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:28.571179 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:28.571649 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:28.571754 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:28.574408 (+ 2654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:56:28.574412 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:56:28.574415 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:56:28.574448 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:56:28.576611 (+ 2163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:56:28.577123 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:56:28.577126 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:56:28.577129 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:56:28.577223 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:56:28.577530 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:56:28.577533 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:56:28.577535 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:56:28.577772 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:56:28.577869 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0924 09:56:28.578126 (+ 257us) server_negotiation.cc:299] Negotiation successful
0924 09:56:28.578331 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":54,"server-negotiator.queue_time_us":48}
I20260924 09:56:28.582926 7281 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c8287837bd048888df868ab81b87c63: Bootstrap starting.
I20260924 09:56:28.586206 7281 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3c8287837bd048888df868ab81b87c63: Neither blocks nor log segments found. Creating new log.
I20260924 09:56:28.587131 7281 log.cc:824] T 00000000000000000000000000000000 P 3c8287837bd048888df868ab81b87c63: Log is configured to *not* fsync() on all Append() calls
I20260924 09:56:28.589494 7281 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c8287837bd048888df868ab81b87c63: No bootstrap required, opened a new log
I20260924 09:56:28.596115 7281 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3c8287837bd048888df868ab81b87c63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c65adc94d5fe46cb8d1244957f570265" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43811 } } peers { permanent_uuid: "b3e4c99bb58342dc915b81f8ea383475" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46619 } } peers { permanent_uuid: "3c8287837bd048888df868ab81b87c63" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34989 } }
I20260924 09:56:28.596422 7281 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3c8287837bd048888df868ab81b87c63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 09:56:28.596527 7281 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3c8287837bd048888df868ab81b87c63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c8287837bd048888df868ab81b87c63, State: Initialized, Role: FOLLOWER
I20260924 09:56:28.597043 7281 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3c8287837bd048888df868ab81b87c63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c65adc94d5fe46cb8d1244957f570265" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43811 } } peers { permanent_uuid: "b3e4c99bb58342dc915b81f8ea383475" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46619 } } peers { permanent_uuid: "3c8287837bd048888df868ab81b87c63" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34989 } }
I20260924 09:56:28.597832 7287 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c8287837bd048888df868ab81b87c63 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c65adc94d5fe46cb8d1244957f570265" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43811 } } peers { permanent_uuid: "b3e4c99bb58342dc915b81f8ea383475" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46619 } } peers { permanent_uuid: "3c8287837bd048888df868ab81b87c63" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34989 } } }
I20260924 09:56:28.598114 7287 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c8287837bd048888df868ab81b87c63 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 09:56:28.598730 7281 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3c8287837bd048888df868ab81b87c63 [sys.catalog]: configured and running, proceeding with master startup.
W20260924 09:56:28.607626 7298 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3c8287837bd048888df868ab81b87c63: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 09:56:28.607823 7298 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 09:56:28.608332 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu pid 6621
I20260924 09:56:28.611312 7122 master_runner.cc:428] Detected that this master 3c8287837bd048888df868ab81b87c63 is joining an existing cluster
I20260924 09:56:28.611419 7122 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.97.60:34989
Sep 24 09:56:28 dist-test-slave-w8d2 krb5kdc[6515](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243785, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.1@KRBTEST.COM
I20260924 09:56:28.629829 7300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:28.609217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53560 (local address 127.0.97.1:46191)
0924 09:56:28.609741 (+ 524us) server_negotiation.cc:206] Beginning negotiation
0924 09:56:28.609750 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 09:56:28.609777 (+ 27us) server_negotiation.cc:407] Connection header received
0924 09:56:28.609839 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:56:28.609847 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:56:28.610100 (+ 253us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:56:28.610308 (+ 208us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:56:28.611858 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:28.613388 (+ 1530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:28.615307 (+ 1919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:28.616466 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:28.620388 (+ 3922us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:28.622156 (+ 1768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:56:28.622169 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:56:28.622184 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:56:28.622224 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:56:28.625933 (+ 3709us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:56:28.626798 (+ 865us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:56:28.626806 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:56:28.626810 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:56:28.626957 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:56:28.627395 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:56:28.627399 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:56:28.627402 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:56:28.627786 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:56:28.629052 (+ 1266us) server_negotiation.cc:1035] Waiting for connection context
0924 09:56:28.629200 (+ 148us) server_negotiation.cc:299] Negotiation successful
0924 09:56:28.629447 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":379,"thread_start_us":233,"threads_started":1}
I20260924 09:56:28.632148 6744 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 09:56:28.637045 6751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:28.615724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52620 (local address 127.0.97.62:43811)
0924 09:56:28.615855 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0924 09:56:28.615860 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 09:56:28.616794 (+ 934us) server_negotiation.cc:407] Connection header received
0924 09:56:28.617063 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:56:28.617068 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:56:28.617148 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:56:28.617221 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:56:28.618613 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:28.619763 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:28.623645 (+ 3882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:28.624292 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:28.624374 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:28.628611 (+ 4237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:56:28.628617 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:56:28.628622 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:56:28.628683 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:56:28.633901 (+ 5218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:56:28.634623 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:56:28.634628 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:56:28.634632 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:56:28.634786 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:56:28.635250 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:56:28.635256 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:56:28.635259 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:56:28.635768 (+ 509us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:56:28.635908 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0924 09:56:28.636509 (+ 601us) server_negotiation.cc:299] Negotiation successful
0924 09:56:28.636719 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 09:56:28.638275 7305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:28.615363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43811 (local address 127.0.0.1:52620)
0924 09:56:28.616689 (+ 1326us) negotiation.cc:107] Waiting for socket to connect
0924 09:56:28.616707 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 09:56:28.616912 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:56:28.617426 (+ 514us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:56:28.617431 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:56:28.618090 (+ 659us) client_negotiation.cc:189] Negotiated authn=SASL
0924 09:56:28.618428 (+ 338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:28.618447 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:28.619937 (+ 1490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:28.619943 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:28.623487 (+ 3544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:28.623500 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:28.626239 (+ 2739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:28.626248 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:28.626663 (+ 415us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:28.627279 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 09:56:28.627298 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 09:56:28.628423 (+ 1125us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 09:56:28.634034 (+ 5611us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 09:56:28.634039 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 09:56:28.634043 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 09:56:28.634492 (+ 449us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 09:56:28.634923 (+ 431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 09:56:28.634936 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 09:56:28.634940 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 09:56:28.635101 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 09:56:28.635975 (+ 874us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 09:56:28.635979 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 09:56:28.636185 (+ 206us) client_negotiation.cc:769] Sending connection context
0924 09:56:28.637508 (+ 1323us) client_negotiation.cc:240] Negotiation successful
0924 09:56:28.637822 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1207,"mutex_wait_us":207,"thread_start_us":1306,"threads_started":1}
I20260924 09:56:28.641892 7213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:28.616504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53212 (local address 127.0.97.61:46619)
0924 09:56:28.616662 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0924 09:56:28.616667 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 09:56:28.626914 (+ 10247us) server_negotiation.cc:407] Connection header received
0924 09:56:28.627079 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:56:28.627084 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:56:28.627163 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:56:28.627278 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:56:28.630137 (+ 2859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:28.631103 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:28.633153 (+ 2050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:28.633765 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:28.633860 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:28.636008 (+ 2148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:56:28.636014 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:56:28.636018 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:56:28.636054 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:56:28.639023 (+ 2969us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:56:28.639771 (+ 748us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:56:28.639775 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:56:28.639778 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:56:28.639910 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:56:28.640340 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:56:28.640345 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:56:28.640347 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:56:28.640696 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:56:28.640816 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0924 09:56:28.641384 (+ 568us) server_negotiation.cc:299] Negotiation successful
0924 09:56:28.641588 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 09:56:28.643409 7306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:28.616950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:46619 (local address 127.0.0.1:53212)
0924 09:56:28.626828 (+ 9878us) negotiation.cc:107] Waiting for socket to connect
0924 09:56:28.626846 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 09:56:28.626956 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:56:28.627435 (+ 479us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:56:28.627440 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:56:28.628074 (+ 634us) client_negotiation.cc:189] Negotiated authn=SASL
0924 09:56:28.628389 (+ 315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:28.628400 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:28.631303 (+ 2903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:28.631308 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:28.633000 (+ 1692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:28.633013 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:28.634128 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:28.634134 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:28.634280 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:28.634810 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 09:56:28.634831 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 09:56:28.635824 (+ 993us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 09:56:28.639175 (+ 3351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 09:56:28.639179 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 09:56:28.639184 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 09:56:28.639651 (+ 467us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 09:56:28.640054 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 09:56:28.640058 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 09:56:28.640062 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 09:56:28.640220 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 09:56:28.640908 (+ 688us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 09:56:28.640912 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 09:56:28.641113 (+ 201us) client_negotiation.cc:769] Sending connection context
0924 09:56:28.642121 (+ 1008us) client_negotiation.cc:240] Negotiation successful
0924 09:56:28.642433 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9727,"mutex_wait_us":118,"thread_start_us":6658,"threads_started":1}
I20260924 09:56:28.651325 7284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:28.616027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48326 (local address 127.0.97.60:34989)
0924 09:56:28.616263 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0924 09:56:28.616269 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 09:56:28.623601 (+ 7332us) server_negotiation.cc:407] Connection header received
0924 09:56:28.623990 (+ 389us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:56:28.623995 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:56:28.624052 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:56:28.624109 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:56:28.625249 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:28.626464 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:28.631304 (+ 4840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:28.632198 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:28.632319 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:28.646156 (+ 13837us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:56:28.646160 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:56:28.646163 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:56:28.646210 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:56:28.648837 (+ 2627us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:56:28.649406 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:56:28.649409 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:56:28.649411 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:56:28.649540 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:56:28.649872 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:56:28.649875 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:56:28.649877 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:56:28.650167 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:56:28.650314 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0924 09:56:28.650629 (+ 315us) server_negotiation.cc:299] Negotiation successful
0924 09:56:28.650941 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":136,"server-negotiator.queue_time_us":48}
I20260924 09:56:28.651363 7307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:28.616436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34989 (local address 127.0.0.1:48326)
0924 09:56:28.623526 (+ 7090us) negotiation.cc:107] Waiting for socket to connect
0924 09:56:28.623545 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 09:56:28.623860 (+ 315us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:56:28.624291 (+ 431us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:56:28.624295 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:56:28.624886 (+ 591us) client_negotiation.cc:189] Negotiated authn=SASL
0924 09:56:28.625116 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:28.625124 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:28.626857 (+ 1733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:28.626862 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:28.629951 (+ 3089us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:28.629964 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:28.638652 (+ 8688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:28.638659 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:28.638826 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:28.645213 (+ 6387us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 09:56:28.645234 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 09:56:28.646016 (+ 782us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 09:56:28.648959 (+ 2943us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 09:56:28.648964 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 09:56:28.648968 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 09:56:28.649297 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 09:56:28.649638 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 09:56:28.649642 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 09:56:28.649646 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 09:56:28.649772 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 09:56:28.650276 (+ 504us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 09:56:28.650281 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 09:56:28.650427 (+ 146us) client_negotiation.cc:769] Sending connection context
0924 09:56:28.650531 (+ 104us) client_negotiation.cc:240] Negotiation successful
0924 09:56:28.650812 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6973,"thread_start_us":10086,"threads_started":1}
Sep 24 09:56:28 dist-test-slave-w8d2 krb5kdc[6515](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243785, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
I20260924 09:56:28.666318 7114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:28.644789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34989 (local address 127.0.0.1:48334)
0924 09:56:28.645041 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0924 09:56:28.645058 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 09:56:28.645170 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:56:28.651986 (+ 6816us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:56:28.651991 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:56:28.652417 (+ 426us) client_negotiation.cc:189] Negotiated authn=SASL
0924 09:56:28.652623 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:28.652634 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:28.653912 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:28.653917 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:28.654761 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:28.654772 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:28.656059 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:28.656066 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:28.656229 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:28.657247 (+ 1018us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 09:56:28.657270 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 09:56:28.659948 (+ 2678us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 09:56:28.663736 (+ 3788us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 09:56:28.663742 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 09:56:28.663746 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 09:56:28.664257 (+ 511us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 09:56:28.664641 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 09:56:28.664646 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 09:56:28.664650 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 09:56:28.664802 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 09:56:28.665362 (+ 560us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 09:56:28.665367 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 09:56:28.665527 (+ 160us) client_negotiation.cc:769] Sending connection context
0924 09:56:28.665663 (+ 136us) client_negotiation.cc:240] Negotiation successful
0924 09:56:28.665950 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145}
I20260924 09:56:28.666363 7284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:28.644998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48334 (local address 127.0.97.60:34989)
0924 09:56:28.651677 (+ 6679us) server_negotiation.cc:206] Beginning negotiation
0924 09:56:28.651681 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 09:56:28.651693 (+ 12us) server_negotiation.cc:407] Connection header received
0924 09:56:28.651756 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:56:28.651760 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:56:28.651840 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:56:28.651920 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:56:28.652788 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:28.653797 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:28.654919 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:28.655841 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:28.657054 (+ 1213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:28.660134 (+ 3080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:56:28.660140 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:56:28.660145 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:56:28.660184 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:56:28.663594 (+ 3410us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:56:28.664378 (+ 784us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:56:28.664383 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:56:28.664387 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:56:28.664517 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:56:28.664910 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:56:28.664914 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:56:28.664918 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:56:28.665252 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:56:28.665390 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0924 09:56:28.665771 (+ 381us) server_negotiation.cc:299] Negotiation successful
0924 09:56:28.665991 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6607,"thread_start_us":10823,"threads_started":1}
I20260924 09:56:28.668510 6559 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.97.60:34989: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "3c8287837bd048888df868ab81b87c63"
member_type: NON_VOTER
last_known_addr {
host: "127.0.97.60"
port: 34989
}
attrs {
promote: true
}
}
dest_uuid: "c65adc94d5fe46cb8d1244957f570265"
cas_config_opid_index: 6
I20260924 09:56:28.669277 6559 consensus_queue.cc:237] T 00000000000000000000000000000000 P c65adc94d5fe46cb8d1244957f570265 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c65adc94d5fe46cb8d1244957f570265" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43811 } } peers { permanent_uuid: "b3e4c99bb58342dc915b81f8ea383475" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46619 } attrs { promote: false } } peers { permanent_uuid: "3c8287837bd048888df868ab81b87c63" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 34989 } attrs { promote: true } }
I20260924 09:56:28.671573 7173 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b3e4c99bb58342dc915b81f8ea383475 [term 1 FOLLOWER]: Refusing update from remote peer c65adc94d5fe46cb8d1244957f570265: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260924 09:56:28.671871 7256 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3c8287837bd048888df868ab81b87c63 [term 0 FOLLOWER]: Advancing to term 1
I20260924 09:56:28.672183 7118 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c65adc94d5fe46cb8d1244957f570265 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3e4c99bb58342dc915b81f8ea383475" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46619 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260924 09:56:28.674710 7118 raft_consensus.cc:2932] T 00000000000000000000000000000000 P c65adc94d5fe46cb8d1244957f570265 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 3c8287837bd048888df868ab81b87c63 (127.0.97.60) added. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c65adc94d5fe46cb8d1244957f570265" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43811 } } peers { permanent_uuid: "b3e4c99bb58342dc915b81f8ea383475" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46619 } attrs { promote: false } } peers { permanent_uuid: "3c8287837bd048888df868ab81b87c63" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 34989 } attrs { promote: true } } }
I20260924 09:56:28.675591 7256 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3c8287837bd048888df868ab81b87c63 [term 1 FOLLOWER]: Refusing update from remote peer c65adc94d5fe46cb8d1244957f570265: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260924 09:56:28.678122 7118 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.97.60:34989
I20260924 09:56:28.678294 7117 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c65adc94d5fe46cb8d1244957f570265 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c8287837bd048888df868ab81b87c63" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 34989 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 09:56:28.679145 7173 raft_consensus.cc:2932] T 00000000000000000000000000000000 P b3e4c99bb58342dc915b81f8ea383475 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 3c8287837bd048888df868ab81b87c63 (127.0.97.60) added. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c65adc94d5fe46cb8d1244957f570265" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43811 } } peers { permanent_uuid: "b3e4c99bb58342dc915b81f8ea383475" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46619 } attrs { promote: false } } peers { permanent_uuid: "3c8287837bd048888df868ab81b87c63" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 34989 } attrs { promote: true } } }
I20260924 09:56:28.681098 7122 master.cc:537] Master@127.0.97.60:34989 shutting down...
I20260924 09:56:28.681013 7119 sys_catalog.cc:455] T 00000000000000000000000000000000 P c65adc94d5fe46cb8d1244957f570265 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "c65adc94d5fe46cb8d1244957f570265" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c65adc94d5fe46cb8d1244957f570265" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43811 } } peers { permanent_uuid: "b3e4c99bb58342dc915b81f8ea383475" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46619 } attrs { promote: false } } peers { permanent_uuid: "3c8287837bd048888df868ab81b87c63" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 34989 } attrs { promote: true } } }
I20260924 09:56:28.681283 7119 sys_catalog.cc:458] T 00000000000000000000000000000000 P c65adc94d5fe46cb8d1244957f570265 [sys.catalog]: This master's current role is: LEADER
I20260924 09:56:28.682680 7119 sys_catalog.cc:455] T 00000000000000000000000000000000 P c65adc94d5fe46cb8d1244957f570265 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c65adc94d5fe46cb8d1244957f570265" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c65adc94d5fe46cb8d1244957f570265" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43811 } } peers { permanent_uuid: "b3e4c99bb58342dc915b81f8ea383475" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46619 } attrs { promote: false } } peers { permanent_uuid: "3c8287837bd048888df868ab81b87c63" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 34989 } attrs { promote: true } } }
I20260924 09:56:28.682883 7119 sys_catalog.cc:458] T 00000000000000000000000000000000 P c65adc94d5fe46cb8d1244957f570265 [sys.catalog]: This master's current role is: LEADER
W20260924 09:56:28.683971 6544 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260924 09:56:28.684314 7201 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3e4c99bb58342dc915b81f8ea383475 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c65adc94d5fe46cb8d1244957f570265" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c65adc94d5fe46cb8d1244957f570265" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43811 } } peers { permanent_uuid: "b3e4c99bb58342dc915b81f8ea383475" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46619 } attrs { promote: false } } peers { permanent_uuid: "3c8287837bd048888df868ab81b87c63" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 34989 } attrs { promote: true } } }
I20260924 09:56:28.684590 7201 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3e4c99bb58342dc915b81f8ea383475 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 09:56:28.693941 7122 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 3c8287837bd048888df868ab81b87c63 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 09:56:28.694432 7122 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 3c8287837bd048888df868ab81b87c63 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 09:56:28.694705 7122 tablet_replica.cc:330] T 00000000000000000000000000000000 P 3c8287837bd048888df868ab81b87c63: stopping tablet replica
W20260924 09:56:28.698421 6544 connection.cc:535] client connection to 127.0.97.60:34989 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260924 09:56:28.698686 6544 consensus_peers.cc:487] T 00000000000000000000000000000000 P c65adc94d5fe46cb8d1244957f570265 -> Peer 3c8287837bd048888df868ab81b87c63 (127.0.97.60:34989): Couldn't send request to peer 3c8287837bd048888df868ab81b87c63. Status: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
I20260924 09:56:28.701330 7122 master.cc:552] Master@127.0.97.60:34989 shutdown complete.
I20260924 09:56:28.701406 7122 master_runner.cc:305] Clearing existing system tablet
I20260924 09:56:28.701820 7122 fs_manager.cc:359] Metadata directory not provided
I20260924 09:56:28.701947 7122 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 09:56:28.704826 7122 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20260924 09:56:28.706339 7312 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 09:56:28.706676 7122 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790243735634835-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 09:56:28.707056 7122 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 09:56:28.707194 7122 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790243735634835-388-0/minicluster-data/master-2/data,/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790243735634835-388-0/minicluster-data/master-2/wal
uuid: "3c8287837bd048888df868ab81b87c63"
format_stamp: "Formatted at 2026-09-24 09:56:28 on dist-test-slave-w8d2"
server_key: "2070cee9a1958a60ab4952e16b2765bb"
server_key_iv: "9dff8533c17ad75ba5c0360e8165b40f"
server_key_version: "encryptionkey@0"
I20260924 09:56:28.712359 7122 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 3c8287837bd048888df868ab81b87c63: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 09:56:28.719264 7122 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 3c8287837bd048888df868ab81b87c63: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260924 09:56:28.719538 7122 log.cc:1196] T 00000000000000000000000000000000 P 3c8287837bd048888df868ab81b87c63: Deleting WAL directory at /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790243735634835-388-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260924 09:56:28.720033 7122 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 3c8287837bd048888df868ab81b87c63: Deleting consensus metadata
I20260924 09:56:28.720314 7122 master_runner.cc:315] Copying system tablet from 127.0.97.62:43811
I20260924 09:56:28.724712 7122 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 3c8287837bd048888df868ab81b87c63: tablet copy: Beginning tablet copy session from remote peer at address 127.0.97.62:43811
I20260924 09:56:28.745949 6751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:28.725604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52634 (local address 127.0.97.62:43811)
0924 09:56:28.725872 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0924 09:56:28.725878 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 09:56:28.728109 (+ 2231us) server_negotiation.cc:407] Connection header received
0924 09:56:28.728354 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:56:28.728360 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:56:28.728436 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:56:28.728512 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:56:28.729673 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:28.730796 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:28.732665 (+ 1869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:28.733322 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:28.733411 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:28.735854 (+ 2443us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:56:28.735860 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:56:28.735866 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:56:28.735902 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:56:28.739223 (+ 3321us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:56:28.740157 (+ 934us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:56:28.740163 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:56:28.740167 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:56:28.740317 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:56:28.743547 (+ 3230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:56:28.743552 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:56:28.743556 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:56:28.744047 (+ 491us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:56:28.744220 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0924 09:56:28.745005 (+ 785us) server_negotiation.cc:299] Negotiation successful
0924 09:56:28.745394 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137}
I20260924 09:56:28.747790 7318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:28.725521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43811 (local address 127.0.0.1:52634)
0924 09:56:28.727994 (+ 2473us) negotiation.cc:107] Waiting for socket to connect
0924 09:56:28.728016 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 09:56:28.728223 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:56:28.728719 (+ 496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:56:28.728724 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:56:28.729246 (+ 522us) client_negotiation.cc:189] Negotiated authn=SASL
0924 09:56:28.729493 (+ 247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:28.729504 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:28.730987 (+ 1483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:28.730992 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:28.732506 (+ 1514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:28.732520 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:28.733694 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:28.733700 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:28.733876 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:28.734557 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 09:56:28.734578 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 09:56:28.735650 (+ 1072us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 09:56:28.739390 (+ 3740us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 09:56:28.739395 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 09:56:28.739403 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0924 09:56:28.739979 (+ 576us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 09:56:28.740428 (+ 449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 09:56:28.740433 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 09:56:28.740437 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 09:56:28.743365 (+ 2928us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 09:56:28.744335 (+ 970us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 09:56:28.744340 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 09:56:28.744601 (+ 261us) client_negotiation.cc:769] Sending connection context
0924 09:56:28.746661 (+ 2060us) client_negotiation.cc:240] Negotiation successful
0924 09:56:28.747176 (+ 515us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1963,"thread_start_us":1843,"threads_started":1}
I20260924 09:56:28.748924 6580 tablet_copy_service.cc:140] P c65adc94d5fe46cb8d1244957f570265: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 3c8287837bd048888df868ab81b87c63 ({username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:52634)
I20260924 09:56:28.749245 6580 tablet_copy_service.cc:161] P c65adc94d5fe46cb8d1244957f570265: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 3c8287837bd048888df868ab81b87c63 at {username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:52634: session id = 3c8287837bd048888df868ab81b87c63-00000000000000000000000000000000
I20260924 09:56:28.753041 6580 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P c65adc94d5fe46cb8d1244957f570265: Tablet Copy: opened 0 blocks and 1 log segments
I20260924 09:56:28.754956 7122 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260924 09:56:28.762290 7122 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 3c8287837bd048888df868ab81b87c63: tablet copy: Starting download of 0 data blocks...
I20260924 09:56:28.762574 7122 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 3c8287837bd048888df868ab81b87c63: tablet copy: Starting download of 1 WAL segments...
I20260924 09:56:28.764907 7122 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 3c8287837bd048888df868ab81b87c63: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260924 09:56:28.769105 6580 tablet_copy_service.cc:342] P c65adc94d5fe46cb8d1244957f570265: Request end of tablet copy session 3c8287837bd048888df868ab81b87c63-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:52634
I20260924 09:56:28.769238 6580 tablet_copy_service.cc:433] P c65adc94d5fe46cb8d1244957f570265: ending tablet copy session 3c8287837bd048888df868ab81b87c63-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 3c8287837bd048888df868ab81b87c63
I20260924 09:56:28.776346 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu with pid 6621
I20260924 09:56:28.779218 7122 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 09:56:28.784806 7319 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:56:28.785243 7320 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:56:28.799582 7322 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 09:56:28.799598 7122 server_base.cc:1034] running on GCE node
I20260924 09:56:28.799837 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
/tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790243735634835-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790243735634835-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790243735634835-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790243735634835-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:46191
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:43811
--webserver_port=38193
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790243735634835-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.97.62:43811,127.0.97.61:46619
--tserver_master_addrs=127.0.97.62:43811,127.0.97.61:46619,127.0.97.60:34989 with env {KRB5CCNAME=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790243735634835-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790243735634835-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790243735634835-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 09:56:28.800114 7122 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 09:56:28.800786 7122 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 09:56:28.801983 7122 hybrid_clock.cc:648] HybridClock initialized: now 1790243788801930 us; error 50 us; skew 500 ppm
I20260924 09:56:28.802217 7122 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 09:56:28 dist-test-slave-w8d2 krb5kdc[6515](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243788, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 09:56:28.804319 7122 init.cc:377] Logged in from keytab as oryx/127.0.97.60@KRBTEST.COM (short username oryx)
I20260924 09:56:28.806128 7122 webserver.cc:460] Webserver started at http://127.0.97.60:44309/ using document root <none> and password file <none>
I20260924 09:56:28.806468 7122 fs_manager.cc:359] Metadata directory not provided
I20260924 09:56:28.806567 7122 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 09:56:28.809743 7122 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20260924 09:56:28.817685 7330 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 09:56:28.818168 7122 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.004s sys 0.004s
I20260924 09:56:28.818288 7122 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790243735634835-388-0/minicluster-data/master-2/data,/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790243735634835-388-0/minicluster-data/master-2/wal
uuid: "3c8287837bd048888df868ab81b87c63"
format_stamp: "Formatted at 2026-09-24 09:56:28 on dist-test-slave-w8d2"
server_key: "2070cee9a1958a60ab4952e16b2765bb"
server_key_iv: "9dff8533c17ad75ba5c0360e8165b40f"
server_key_version: "encryptionkey@0"
I20260924 09:56:28.818471 7122 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790243735634835-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790243735634835-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790243735634835-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 09:56:28.829914 7122 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 09:56:28.833856 7122 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 09:56:28.862116 7122 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:34989
I20260924 09:56:28.862164 7381 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:34989 every 8 connection(s)
I20260924 09:56:28.863112 7122 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790243735634835-388-0/minicluster-data/master-2/data/info.pb
I20260924 09:56:28.868165 7382 sys_catalog.cc:263] Verifying existing consensus state
I20260924 09:56:28.871889 7382 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c8287837bd048888df868ab81b87c63: Bootstrap starting.
I20260924 09:56:28.888527 7382 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c8287837bd048888df868ab81b87c63: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 09:56:28.889029 7382 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c8287837bd048888df868ab81b87c63: Bootstrap complete.
I20260924 09:56:28.890110 7382 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3c8287837bd048888df868ab81b87c63 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c65adc94d5fe46cb8d1244957f570265" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43811 } } peers { permanent_uuid: "b3e4c99bb58342dc915b81f8ea383475" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46619 } attrs { promote: false } } peers { permanent_uuid: "3c8287837bd048888df868ab81b87c63" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 34989 } attrs { promote: true } }
I20260924 09:56:28.890417 7382 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3c8287837bd048888df868ab81b87c63 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3c8287837bd048888df868ab81b87c63, State: Initialized, Role: LEARNER
I20260924 09:56:28.890669 7382 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3c8287837bd048888df868ab81b87c63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c65adc94d5fe46cb8d1244957f570265" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43811 } } peers { permanent_uuid: "b3e4c99bb58342dc915b81f8ea383475" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46619 } attrs { promote: false } } peers { permanent_uuid: "3c8287837bd048888df868ab81b87c63" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 34989 } attrs { promote: true } }
I20260924 09:56:28.892052 7382 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3c8287837bd048888df868ab81b87c63 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 09:56:28.891782 7385 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c8287837bd048888df868ab81b87c63 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c65adc94d5fe46cb8d1244957f570265" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43811 } } peers { permanent_uuid: "b3e4c99bb58342dc915b81f8ea383475" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46619 } attrs { promote: false } } peers { permanent_uuid: "3c8287837bd048888df868ab81b87c63" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 34989 } attrs { promote: true } } }
I20260924 09:56:28.892300 7385 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c8287837bd048888df868ab81b87c63 [sys.catalog]: This master's current role is: LEARNER
I20260924 09:56:28.926456 7396 catalog_manager.cc:1259] Loaded cluster ID: 0d84da8a40b44bcdbd4e55f25ff820dc
I20260924 09:56:28.926599 7396 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 3c8287837bd048888df868ab81b87c63: loading cluster ID for follower catalog manager: success
I20260924 09:56:28.930003 7396 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 3c8287837bd048888df868ab81b87c63: acquiring CA information for follower catalog manager: success
I20260924 09:56:28.932492 7396 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 3c8287837bd048888df868ab81b87c63: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 09:56:29.130501 7114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:29.124083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34989 (local address 127.0.0.1:48344)
0924 09:56:29.124265 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0924 09:56:29.124287 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 09:56:29.124425 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:56:29.125019 (+ 594us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:56:29.125024 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:56:29.125052 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 09:56:29.125226 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:29.125261 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:29.126502 (+ 1241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:29.126508 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:29.128371 (+ 1863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:29.128386 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:29.129757 (+ 1371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:29.129763 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:29.129958 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:29.129996 (+ 38us) client_negotiation.cc:769] Sending connection context
0924 09:56:29.130079 (+ 83us) client_negotiation.cc:240] Negotiation successful
0924 09:56:29.130214 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260924 09:56:29.130663 7397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:29.124252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48344 (local address 127.0.97.60:34989)
0924 09:56:29.124735 (+ 483us) server_negotiation.cc:206] Beginning negotiation
0924 09:56:29.124739 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 09:56:29.124754 (+ 15us) server_negotiation.cc:407] Connection header received
0924 09:56:29.124809 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:56:29.124812 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:56:29.124873 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:56:29.124971 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 09:56:29.125405 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:29.126359 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:29.128525 (+ 2166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:29.129567 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:29.129701 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:29.130285 (+ 584us) server_negotiation.cc:1035] Waiting for connection context
0924 09:56:29.130327 (+ 42us) server_negotiation.cc:299] Negotiation successful
0924 09:56:29.130417 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":392,"thread_start_us":268,"threads_started":1}
I20260924 09:56:29.131644 7357 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 3c8287837bd048888df868ab81b87c63 [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.7] Dedup: 1.7->[]
I20260924 09:56:29.132172 7385 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c8287837bd048888df868ab81b87c63 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c65adc94d5fe46cb8d1244957f570265. Latest consensus state: current_term: 1 leader_uuid: "c65adc94d5fe46cb8d1244957f570265" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c65adc94d5fe46cb8d1244957f570265" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43811 } } peers { permanent_uuid: "b3e4c99bb58342dc915b81f8ea383475" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46619 } attrs { promote: false } } peers { permanent_uuid: "3c8287837bd048888df868ab81b87c63" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 34989 } attrs { promote: true } } }
I20260924 09:56:29.132418 7385 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c8287837bd048888df868ab81b87c63 [sys.catalog]: This master's current role is: LEARNER
W20260924 09:56:29.148911 7327 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 09:56:29.149165 7327 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 09:56:29.149296 7327 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 09:56:29.156205 7327 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 09:56:29.156313 7327 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 09:56:29.156512 7327 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260924 09:56:29.166265 7327 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790243735634835-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790243735634835-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790243735634835-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:46191
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790243735634835-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=38193
--tserver_master_addrs=127.0.97.62:43811,127.0.97.61:46619,127.0.97.60:34989
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790243735634835-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 09:56:29.167622 7327 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 09:56:29.168875 7327 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 09:56:29.181943 7402 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:56:29.182147 7403 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:56:29.183588 7405 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 09:56:29.184063 7327 server_base.cc:1034] running on GCE node
I20260924 09:56:29.184643 7327 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 09:56:29.185546 7327 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 09:56:29.186753 7327 hybrid_clock.cc:648] HybridClock initialized: now 1790243789186690 us; error 52 us; skew 500 ppm
I20260924 09:56:29.187069 7327 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 09:56:29 dist-test-slave-w8d2 krb5kdc[6515](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243789, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 09:56:29.189963 7327 init.cc:377] Logged in from keytab as oryx/127.0.97.1@KRBTEST.COM (short username oryx)
I20260924 09:56:29.191355 7327 webserver.cc:460] Webserver started at http://127.0.97.1:38193/ using document root <none> and password file <none>
I20260924 09:56:29.192087 7327 fs_manager.cc:359] Metadata directory not provided
I20260924 09:56:29.192203 7327 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 09:56:29.197322 7327 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.003s
I20260924 09:56:29.200260 7412 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 09:56:29.201495 7327 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260924 09:56:29.201627 7327 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790243735634835-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790243735634835-388-0/minicluster-data/ts-0/wal
uuid: "87e95d359d14496285f25e40421c70c8"
format_stamp: "Formatted at 2026-09-24 09:56:26 on dist-test-slave-w8d2"
server_key: "682627180f64f6ab86ad61cbf8a2fd93"
server_key_iv: "1bf7ed6f2cccf0bf3288b0eba6064106"
server_key_version: "encryptionkey@0"
I20260924 09:56:29.202500 7327 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790243735634835-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790243735634835-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790243735634835-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 09:56:29.224037 7327 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 09:56:29.227813 7327 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 09:56:29.228056 7327 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 09:56:29.228955 7327 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 09:56:29.230218 7327 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 09:56:29.230307 7327 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 09:56:29.230408 7327 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 09:56:29.230489 7327 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 09:56:29.260749 7327 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:46191
I20260924 09:56:29.260813 7524 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:46191 every 8 connection(s)
I20260924 09:56:29.262081 7327 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790243735634835-388-0/minicluster-data/ts-0/data/info.pb
I20260924 09:56:29.270851 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu as pid 7327
I20260924 09:56:29.271121 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu pid 6755
Sep 24 09:56:29 dist-test-slave-w8d2 krb5kdc[6515](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243789, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
Sep 24 09:56:29 dist-test-slave-w8d2 krb5kdc[6515](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243789, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
Sep 24 09:56:29 dist-test-slave-w8d2 krb5kdc[6515](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243789, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260924 09:56:29.290186 7397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:29.264874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:45355 (local address 127.0.97.60:34989)
0924 09:56:29.265062 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0924 09:56:29.265068 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 09:56:29.266847 (+ 1779us) server_negotiation.cc:407] Connection header received
0924 09:56:29.269664 (+ 2817us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:56:29.269671 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:56:29.269763 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:56:29.269875 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:56:29.273263 (+ 3388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:29.274378 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:29.276919 (+ 2541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:29.277526 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:29.277643 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:29.283094 (+ 5451us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:56:29.283101 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:56:29.283107 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:56:29.283161 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:56:29.286545 (+ 3384us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:56:29.287554 (+ 1009us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:56:29.287559 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:56:29.287563 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:56:29.287702 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:56:29.288136 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:56:29.288141 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:56:29.288143 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:56:29.288469 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:56:29.288579 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0924 09:56:29.289628 (+ 1049us) server_negotiation.cc:299] Negotiation successful
0924 09:56:29.289860 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 09:56:29.290200 7529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:29.264874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:42669 (local address 127.0.97.62:43811)
0924 09:56:29.265906 (+ 1032us) server_negotiation.cc:206] Beginning negotiation
0924 09:56:29.265912 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 09:56:29.270081 (+ 4169us) server_negotiation.cc:407] Connection header received
0924 09:56:29.270211 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:56:29.270216 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:56:29.270290 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:56:29.270381 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:56:29.273238 (+ 2857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:29.274345 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:29.276636 (+ 2291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:29.277132 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:29.277251 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:29.283082 (+ 5831us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:56:29.283089 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:56:29.283094 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:56:29.283132 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:56:29.286546 (+ 3414us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:56:29.287468 (+ 922us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:56:29.287473 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:56:29.287477 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:56:29.288118 (+ 641us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:56:29.288490 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:56:29.288495 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:56:29.288498 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:56:29.288804 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:56:29.288919 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0924 09:56:29.289618 (+ 699us) server_negotiation.cc:299] Negotiation successful
0924 09:56:29.289833 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":918,"thread_start_us":1030,"threads_started":1}
I20260924 09:56:29.291033 7530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:29.264923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:43511 (local address 127.0.97.61:46619)
0924 09:56:29.266267 (+ 1344us) server_negotiation.cc:206] Beginning negotiation
0924 09:56:29.266274 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 09:56:29.268312 (+ 2038us) server_negotiation.cc:407] Connection header received
0924 09:56:29.270023 (+ 1711us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:56:29.270027 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:56:29.270112 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:56:29.270209 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:56:29.273238 (+ 3029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:29.274345 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:29.278277 (+ 3932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:29.278835 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:29.278956 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:29.283082 (+ 4126us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:56:29.283089 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:56:29.283094 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:56:29.283132 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:56:29.286277 (+ 3145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:56:29.287356 (+ 1079us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:56:29.287361 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:56:29.287364 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:56:29.287499 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:56:29.288140 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:56:29.288144 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:56:29.288147 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:56:29.288473 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:56:29.288623 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0924 09:56:29.290488 (+ 1865us) server_negotiation.cc:299] Negotiation successful
0924 09:56:29.290696 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1220,"thread_start_us":1096,"threads_started":1}
I20260924 09:56:29.291901 7533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:29.265584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43811 (local address 127.0.97.1:42669)
0924 09:56:29.269977 (+ 4393us) negotiation.cc:107] Waiting for socket to connect
0924 09:56:29.269997 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 09:56:29.270103 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:56:29.270413 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:56:29.270417 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:56:29.272072 (+ 1655us) client_negotiation.cc:189] Negotiated authn=SASL
0924 09:56:29.273042 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:29.273053 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:29.274517 (+ 1464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:29.274522 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:29.276499 (+ 1977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:29.276510 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:29.277522 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:29.277530 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:29.277821 (+ 291us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:29.278546 (+ 725us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 09:56:29.278575 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0924 09:56:29.282876 (+ 4301us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 09:56:29.286690 (+ 3814us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 09:56:29.286694 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 09:56:29.286697 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 09:56:29.287351 (+ 654us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 09:56:29.288243 (+ 892us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 09:56:29.288248 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 09:56:29.288251 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 09:56:29.288372 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 09:56:29.288986 (+ 614us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 09:56:29.288990 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 09:56:29.289196 (+ 206us) client_negotiation.cc:769] Sending connection context
0924 09:56:29.290374 (+ 1178us) client_negotiation.cc:240] Negotiation successful
0924 09:56:29.290627 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4254,"mutex_wait_us":626,"spinlock_wait_cycles":44160,"thread_start_us":3768,"threads_started":1}
I20260924 09:56:29.291993 7531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:29.265583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:46619 (local address 127.0.97.1:43511)
0924 09:56:29.268184 (+ 2601us) negotiation.cc:107] Waiting for socket to connect
0924 09:56:29.268206 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 09:56:29.269364 (+ 1158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:56:29.270241 (+ 877us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:56:29.270245 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:56:29.272077 (+ 1832us) client_negotiation.cc:189] Negotiated authn=SASL
0924 09:56:29.273039 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:29.273053 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:29.274525 (+ 1472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:29.274529 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:29.278157 (+ 3628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:29.278168 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:29.279590 (+ 1422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:29.279598 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:29.279784 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:29.280396 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 09:56:29.280420 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 09:56:29.282876 (+ 2456us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 09:56:29.286466 (+ 3590us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 09:56:29.286476 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 09:56:29.286495 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0924 09:56:29.287177 (+ 682us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 09:56:29.287650 (+ 473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 09:56:29.287656 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 09:56:29.287659 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 09:56:29.288010 (+ 351us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 09:56:29.288679 (+ 669us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 09:56:29.288683 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 09:56:29.289196 (+ 513us) client_negotiation.cc:769] Sending connection context
0924 09:56:29.289505 (+ 309us) client_negotiation.cc:240] Negotiation successful
0924 09:56:29.290004 (+ 499us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1979,"mutex_wait_us":452,"spinlock_wait_cycles":21888,"thread_start_us":497,"threads_started":1}
I20260924 09:56:29.291939 7532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:29.265583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34989 (local address 127.0.97.1:45355)
0924 09:56:29.266666 (+ 1083us) negotiation.cc:107] Waiting for socket to connect
0924 09:56:29.266713 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0924 09:56:29.269350 (+ 2637us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:56:29.270159 (+ 809us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:56:29.270171 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:56:29.271988 (+ 1817us) client_negotiation.cc:189] Negotiated authn=SASL
0924 09:56:29.273022 (+ 1034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:29.273049 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:29.274535 (+ 1486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:29.274540 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:29.276769 (+ 2229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:29.276785 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:29.277864 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:29.277870 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:29.278038 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:29.278671 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 09:56:29.278695 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 09:56:29.282810 (+ 4115us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 09:56:29.286709 (+ 3899us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 09:56:29.286714 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 09:56:29.286719 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 09:56:29.287420 (+ 701us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 09:56:29.287839 (+ 419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 09:56:29.287842 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 09:56:29.287845 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 09:56:29.288012 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 09:56:29.288664 (+ 652us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 09:56:29.288671 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 09:56:29.289203 (+ 532us) client_negotiation.cc:769] Sending connection context
0924 09:56:29.290885 (+ 1682us) client_negotiation.cc:240] Negotiation successful
0924 09:56:29.291414 (+ 529us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":773,"mutex_wait_us":162,"spinlock_wait_cycles":56064,"thread_start_us":1699,"threads_started":1}
I20260924 09:56:29.293593 7525 heartbeater.cc:347] Connected to a master server at 127.0.97.60:34989
I20260924 09:56:29.293630 7527 heartbeater.cc:347] Connected to a master server at 127.0.97.61:46619
I20260924 09:56:29.293684 7526 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43811
I20260924 09:56:29.293942 7526 heartbeater.cc:464] Registering TS with master...
I20260924 09:56:29.293946 7525 heartbeater.cc:464] Registering TS with master...
I20260924 09:56:29.293946 7527 heartbeater.cc:464] Registering TS with master...
I20260924 09:56:29.294922 7527 heartbeater.cc:511] Master 127.0.97.61:46619 requested a full tablet report, sending...
I20260924 09:56:29.294947 7525 heartbeater.cc:511] Master 127.0.97.60:34989 requested a full tablet report, sending...
I20260924 09:56:29.295919 7526 heartbeater.cc:511] Master 127.0.97.62:43811 requested a full tablet report, sending...
I20260924 09:56:29.296976 6559 ts_manager.cc:194] Re-registered known tserver with Master: 87e95d359d14496285f25e40421c70c8 (127.0.97.1:46191)
I20260924 09:56:29.297338 7347 ts_manager.cc:194] Registered new tserver with Master: 87e95d359d14496285f25e40421c70c8 (127.0.97.1:46191)
I20260924 09:56:29.297480 7163 ts_manager.cc:194] Registered new tserver with Master: 87e95d359d14496285f25e40421c70c8 (127.0.97.1:46191)
I20260924 09:56:29.298092 6559 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:42669
Sep 24 09:56:29 dist-test-slave-w8d2 krb5kdc[6515](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243785, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.2@KRBTEST.COM
I20260924 09:56:29.306270 7535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:29.272273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56608 (local address 127.0.97.2:35061)
0924 09:56:29.287848 (+ 15575us) server_negotiation.cc:206] Beginning negotiation
0924 09:56:29.287857 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 09:56:29.287887 (+ 30us) server_negotiation.cc:407] Connection header received
0924 09:56:29.287954 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:56:29.287962 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:56:29.288257 (+ 295us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:56:29.290556 (+ 2299us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:56:29.290806 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:29.292566 (+ 1760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:29.294337 (+ 1771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:29.295369 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:29.295505 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:29.300913 (+ 5408us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:56:29.300925 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:56:29.300937 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:56:29.300973 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:56:29.303713 (+ 2740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:56:29.304431 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:56:29.304438 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:56:29.304440 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:56:29.304585 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:56:29.304865 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:56:29.304868 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:56:29.304870 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:56:29.305236 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:56:29.305438 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0924 09:56:29.305706 (+ 268us) server_negotiation.cc:299] Negotiation successful
0924 09:56:29.305897 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15383,"thread_start_us":16116,"threads_started":1}
I20260924 09:56:29.307649 6878 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 09:56:29.390551 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu with pid 6755
I20260924 09:56:29.403278 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
/tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790243735634835-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790243735634835-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790243735634835-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790243735634835-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:35061
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:43811
--webserver_port=38369
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790243735634835-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.97.62:43811,127.0.97.61:46619
--tserver_master_addrs=127.0.97.62:43811,127.0.97.61:46619,127.0.97.60:34989 with env {KRB5CCNAME=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790243735634835-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790243735634835-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790243735634835-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 09:56:29.508065 7119 sys_catalog.cc:455] T 00000000000000000000000000000000 P c65adc94d5fe46cb8d1244957f570265 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c65adc94d5fe46cb8d1244957f570265" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c65adc94d5fe46cb8d1244957f570265" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43811 } } peers { permanent_uuid: "b3e4c99bb58342dc915b81f8ea383475" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46619 } attrs { promote: false } } peers { permanent_uuid: "3c8287837bd048888df868ab81b87c63" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 34989 } attrs { promote: true } } }
I20260924 09:56:29.508437 7119 sys_catalog.cc:458] T 00000000000000000000000000000000 P c65adc94d5fe46cb8d1244957f570265 [sys.catalog]: This master's current role is: LEADER
I20260924 09:56:29.509959 7538 raft_consensus.cc:1050] T 00000000000000000000000000000000 P c65adc94d5fe46cb8d1244957f570265: attempting to promote NON_VOTER 3c8287837bd048888df868ab81b87c63 to VOTER
I20260924 09:56:29.510493 7538 consensus_queue.cc:237] T 00000000000000000000000000000000 P c65adc94d5fe46cb8d1244957f570265 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c65adc94d5fe46cb8d1244957f570265" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43811 } } peers { permanent_uuid: "b3e4c99bb58342dc915b81f8ea383475" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46619 } attrs { promote: false } } peers { permanent_uuid: "3c8287837bd048888df868ab81b87c63" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34989 } attrs { promote: false } }
I20260924 09:56:29.512177 7173 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b3e4c99bb58342dc915b81f8ea383475 [term 1 FOLLOWER]: Refusing update from remote peer c65adc94d5fe46cb8d1244957f570265: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260924 09:56:29.512203 7357 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3c8287837bd048888df868ab81b87c63 [term 1 LEARNER]: Refusing update from remote peer c65adc94d5fe46cb8d1244957f570265: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260924 09:56:29.512713 7539 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c65adc94d5fe46cb8d1244957f570265 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c8287837bd048888df868ab81b87c63" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34989 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260924 09:56:29.513087 7119 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c65adc94d5fe46cb8d1244957f570265 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3e4c99bb58342dc915b81f8ea383475" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46619 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260924 09:56:29.515703 7119 raft_consensus.cc:2932] T 00000000000000000000000000000000 P c65adc94d5fe46cb8d1244957f570265 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 3c8287837bd048888df868ab81b87c63 (127.0.97.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c65adc94d5fe46cb8d1244957f570265" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43811 } } peers { permanent_uuid: "b3e4c99bb58342dc915b81f8ea383475" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46619 } attrs { promote: false } } peers { permanent_uuid: "3c8287837bd048888df868ab81b87c63" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34989 } attrs { promote: false } } }
I20260924 09:56:29.519631 7119 sys_catalog.cc:455] T 00000000000000000000000000000000 P c65adc94d5fe46cb8d1244957f570265 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "c65adc94d5fe46cb8d1244957f570265" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c65adc94d5fe46cb8d1244957f570265" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43811 } } peers { permanent_uuid: "b3e4c99bb58342dc915b81f8ea383475" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46619 } attrs { promote: false } } peers { permanent_uuid: "3c8287837bd048888df868ab81b87c63" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34989 } attrs { promote: false } } }
I20260924 09:56:29.519872 7119 sys_catalog.cc:458] T 00000000000000000000000000000000 P c65adc94d5fe46cb8d1244957f570265 [sys.catalog]: This master's current role is: LEADER
I20260924 09:56:29.520087 7119 sys_catalog.cc:455] T 00000000000000000000000000000000 P c65adc94d5fe46cb8d1244957f570265 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c65adc94d5fe46cb8d1244957f570265" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c65adc94d5fe46cb8d1244957f570265" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43811 } } peers { permanent_uuid: "b3e4c99bb58342dc915b81f8ea383475" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46619 } attrs { promote: false } } peers { permanent_uuid: "3c8287837bd048888df868ab81b87c63" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34989 } attrs { promote: false } } }
I20260924 09:56:29.519898 7173 raft_consensus.cc:2932] T 00000000000000000000000000000000 P b3e4c99bb58342dc915b81f8ea383475 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 3c8287837bd048888df868ab81b87c63 (127.0.97.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c65adc94d5fe46cb8d1244957f570265" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43811 } } peers { permanent_uuid: "b3e4c99bb58342dc915b81f8ea383475" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46619 } attrs { promote: false } } peers { permanent_uuid: "3c8287837bd048888df868ab81b87c63" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34989 } attrs { promote: false } } }
I20260924 09:56:29.520292 7119 sys_catalog.cc:458] T 00000000000000000000000000000000 P c65adc94d5fe46cb8d1244957f570265 [sys.catalog]: This master's current role is: LEADER
I20260924 09:56:29.524612 7542 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3e4c99bb58342dc915b81f8ea383475 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c65adc94d5fe46cb8d1244957f570265" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c65adc94d5fe46cb8d1244957f570265" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43811 } } peers { permanent_uuid: "b3e4c99bb58342dc915b81f8ea383475" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46619 } attrs { promote: false } } peers { permanent_uuid: "3c8287837bd048888df868ab81b87c63" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34989 } attrs { promote: false } } }
I20260924 09:56:29.524997 7542 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3e4c99bb58342dc915b81f8ea383475 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 09:56:29.525185 7539 sys_catalog.cc:455] T 00000000000000000000000000000000 P c65adc94d5fe46cb8d1244957f570265 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c65adc94d5fe46cb8d1244957f570265" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c65adc94d5fe46cb8d1244957f570265" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43811 } } peers { permanent_uuid: "b3e4c99bb58342dc915b81f8ea383475" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46619 } attrs { promote: false } } peers { permanent_uuid: "3c8287837bd048888df868ab81b87c63" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34989 } attrs { promote: false } } }
I20260924 09:56:29.525440 7539 sys_catalog.cc:458] T 00000000000000000000000000000000 P c65adc94d5fe46cb8d1244957f570265 [sys.catalog]: This master's current role is: LEADER
I20260924 09:56:29.525938 7357 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 3c8287837bd048888df868ab81b87c63 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 3c8287837bd048888df868ab81b87c63 (127.0.97.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c65adc94d5fe46cb8d1244957f570265" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43811 } } peers { permanent_uuid: "b3e4c99bb58342dc915b81f8ea383475" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46619 } attrs { promote: false } } peers { permanent_uuid: "3c8287837bd048888df868ab81b87c63" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34989 } attrs { promote: false } } }
I20260924 09:56:29.528815 7385 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c8287837bd048888df868ab81b87c63 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c65adc94d5fe46cb8d1244957f570265" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c65adc94d5fe46cb8d1244957f570265" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43811 } } peers { permanent_uuid: "b3e4c99bb58342dc915b81f8ea383475" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46619 } attrs { promote: false } } peers { permanent_uuid: "3c8287837bd048888df868ab81b87c63" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34989 } attrs { promote: false } } }
I20260924 09:56:29.529100 7385 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c8287837bd048888df868ab81b87c63 [sys.catalog]: This master's current role is: FOLLOWER
W20260924 09:56:29.714910 7537 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 09:56:29.715195 7537 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 09:56:29.715358 7537 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 09:56:29.722561 7537 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 09:56:29.722668 7537 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 09:56:29.722786 7537 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260924 09:56:29.732352 7537 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790243735634835-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790243735634835-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790243735634835-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:35061
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790243735634835-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=38369
--tserver_master_addrs=127.0.97.62:43811,127.0.97.61:46619,127.0.97.60:34989
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790243735634835-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 09:56:29.733551 7537 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 09:56:29.734745 7537 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 09:56:29.746974 7547 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:56:29.747045 7548 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:56:29.747046 7550 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 09:56:29.747601 7537 server_base.cc:1034] running on GCE node
I20260924 09:56:29.748081 7537 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 09:56:29.748849 7537 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 09:56:29.750032 7537 hybrid_clock.cc:648] HybridClock initialized: now 1790243789749984 us; error 45 us; skew 500 ppm
I20260924 09:56:29.750308 7537 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 09:56:29 dist-test-slave-w8d2 krb5kdc[6515](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243789, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 09:56:29.753136 7537 init.cc:377] Logged in from keytab as oryx/127.0.97.2@KRBTEST.COM (short username oryx)
I20260924 09:56:29.754455 7537 webserver.cc:460] Webserver started at http://127.0.97.2:38369/ using document root <none> and password file <none>
I20260924 09:56:29.755074 7537 fs_manager.cc:359] Metadata directory not provided
I20260924 09:56:29.755190 7537 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 09:56:29.760242 7537 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260924 09:56:29.762694 7557 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 09:56:29.763947 7537 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260924 09:56:29.764086 7537 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790243735634835-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790243735634835-388-0/minicluster-data/ts-1/wal
uuid: "3dc89c3746aa49159412f76b46758c66"
format_stamp: "Formatted at 2026-09-24 09:56:26 on dist-test-slave-w8d2"
server_key: "cca81d8bfd33aa04c3e710b4c31e0242"
server_key_iv: "0d1263420cf2f239589705a7aa1e6fb8"
server_key_version: "encryptionkey@0"
I20260924 09:56:29.764987 7537 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790243735634835-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790243735634835-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790243735634835-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 09:56:29.782608 7537 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 09:56:29.786116 7537 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 09:56:29.786348 7537 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 09:56:29.787289 7537 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 09:56:29.788522 7537 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 09:56:29.788604 7537 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 09:56:29.788703 7537 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 09:56:29.788767 7537 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 09:56:29.817448 7537 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:35061
I20260924 09:56:29.817497 7669 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:35061 every 8 connection(s)
I20260924 09:56:29.818773 7537 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790243735634835-388-0/minicluster-data/ts-1/data/info.pb
I20260924 09:56:29.821007 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu as pid 7537
I20260924 09:56:29.821280 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu pid 6888
Sep 24 09:56:29 dist-test-slave-w8d2 krb5kdc[6515](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243789, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
Sep 24 09:56:29 dist-test-slave-w8d2 krb5kdc[6515](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243785, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.3@KRBTEST.COM
Sep 24 09:56:29 dist-test-slave-w8d2 krb5kdc[6515](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243789, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260924 09:56:29.856909 7678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:29.822702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60314 (local address 127.0.97.3:33087)
0924 09:56:29.827701 (+ 4999us) server_negotiation.cc:206] Beginning negotiation
0924 09:56:29.827711 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 09:56:29.835195 (+ 7484us) server_negotiation.cc:407] Connection header received
0924 09:56:29.835504 (+ 309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:56:29.835514 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:56:29.835796 (+ 282us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:56:29.835921 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:56:29.838056 (+ 2135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:29.839697 (+ 1641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:29.841572 (+ 1875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:29.842445 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:29.842586 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:29.849031 (+ 6445us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:56:29.849048 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:56:29.849065 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:56:29.849127 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:56:29.852775 (+ 3648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:56:29.853998 (+ 1223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:56:29.854007 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:56:29.854010 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:56:29.854149 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:56:29.854830 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:56:29.854835 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:56:29.854838 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:56:29.855469 (+ 631us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:56:29.855770 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0924 09:56:29.856292 (+ 522us) server_negotiation.cc:299] Negotiation successful
0924 09:56:29.856535 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4818,"thread_start_us":4683,"threads_started":1}
I20260924 09:56:29.856954 7677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:29.822108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34989 (local address 127.0.97.2:34439)
0924 09:56:29.831802 (+ 9694us) negotiation.cc:107] Waiting for socket to connect
0924 09:56:29.831843 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 09:56:29.833639 (+ 1796us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:56:29.835209 (+ 1570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:56:29.835223 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:56:29.836104 (+ 881us) client_negotiation.cc:189] Negotiated authn=SASL
0924 09:56:29.837302 (+ 1198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:29.837331 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:29.838989 (+ 1658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:29.838994 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:29.841060 (+ 2066us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:29.841076 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:29.842254 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:29.842261 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:29.842524 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:29.843373 (+ 849us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 09:56:29.843402 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0924 09:56:29.848127 (+ 4725us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 09:56:29.852152 (+ 4025us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 09:56:29.852162 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 09:56:29.852181 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0924 09:56:29.852760 (+ 579us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 09:56:29.853211 (+ 451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 09:56:29.853215 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 09:56:29.853218 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 09:56:29.853493 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 09:56:29.854173 (+ 680us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 09:56:29.854181 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 09:56:29.854607 (+ 426us) client_negotiation.cc:769] Sending connection context
0924 09:56:29.854923 (+ 316us) client_negotiation.cc:240] Negotiation successful
0924 09:56:29.855522 (+ 599us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9429,"thread_start_us":11848,"threads_started":1}
Sep 24 09:56:29 dist-test-slave-w8d2 krb5kdc[6515](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243789, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260924 09:56:29.858281 7011 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 09:56:29.859766 7674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:29.821468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:34439 (local address 127.0.97.60:34989)
0924 09:56:29.828026 (+ 6558us) server_negotiation.cc:206] Beginning negotiation
0924 09:56:29.828030 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 09:56:29.831971 (+ 3941us) server_negotiation.cc:407] Connection header received
0924 09:56:29.833916 (+ 1945us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:56:29.833921 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:56:29.834009 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:56:29.834117 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:56:29.837558 (+ 3441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:29.838809 (+ 1251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:29.841251 (+ 2442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:29.841828 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:29.841947 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:29.848393 (+ 6446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:56:29.848400 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:56:29.848405 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:56:29.848446 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:56:29.851991 (+ 3545us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:56:29.852909 (+ 918us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:56:29.852914 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:56:29.852917 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:56:29.853085 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:56:29.853621 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:56:29.853627 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:56:29.853630 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:56:29.853961 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:56:29.854104 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0924 09:56:29.859172 (+ 5068us) server_negotiation.cc:299] Negotiation successful
0924 09:56:29.859411 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6464,"thread_start_us":6349,"threads_started":1}
I20260924 09:56:29.860790 7679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:29.822845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:42113 (local address 127.0.97.61:46619)
0924 09:56:29.834661 (+ 11816us) server_negotiation.cc:206] Beginning negotiation
0924 09:56:29.834667 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 09:56:29.840632 (+ 5965us) server_negotiation.cc:407] Connection header received
0924 09:56:29.841115 (+ 483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:56:29.841121 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:56:29.841228 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:56:29.841305 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:56:29.843364 (+ 2059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:29.844593 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:29.848809 (+ 4216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:29.849390 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:29.849485 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:29.854303 (+ 4818us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:56:29.854309 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:56:29.854314 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:56:29.854363 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:56:29.857367 (+ 3004us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:56:29.858542 (+ 1175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:56:29.858547 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:56:29.858551 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:56:29.858682 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:56:29.859126 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:56:29.859133 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:56:29.859137 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:56:29.859540 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:56:29.859685 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0924 09:56:29.860160 (+ 475us) server_negotiation.cc:299] Negotiation successful
0924 09:56:29.860408 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11699,"thread_start_us":11610,"threads_started":1}
I20260924 09:56:29.861050 7670 heartbeater.cc:347] Connected to a master server at 127.0.97.60:34989
I20260924 09:56:29.861418 7670 heartbeater.cc:464] Registering TS with master...
I20260924 09:56:29.861743 7675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:29.822108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:46619 (local address 127.0.97.2:42113)
0924 09:56:29.840513 (+ 18405us) negotiation.cc:107] Waiting for socket to connect
0924 09:56:29.840538 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0924 09:56:29.840831 (+ 293us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:56:29.841526 (+ 695us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:56:29.841531 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:56:29.842147 (+ 616us) client_negotiation.cc:189] Negotiated authn=SASL
0924 09:56:29.843102 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:29.843114 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:29.844772 (+ 1658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:29.844778 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:29.848620 (+ 3842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:29.848635 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:29.849766 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:29.849772 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:29.849932 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:29.850435 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 09:56:29.850455 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 09:56:29.854104 (+ 3649us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 09:56:29.857933 (+ 3829us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 09:56:29.857938 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 09:56:29.857942 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 09:56:29.858413 (+ 471us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 09:56:29.858829 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 09:56:29.858833 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 09:56:29.858837 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 09:56:29.858995 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 09:56:29.859747 (+ 752us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 09:56:29.859752 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 09:56:29.859895 (+ 143us) client_negotiation.cc:769] Sending connection context
0924 09:56:29.860952 (+ 1057us) client_negotiation.cc:240] Negotiation successful
0924 09:56:29.861263 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12090,"mutex_wait_us":283,"thread_start_us":9027,"threads_started":1}
I20260924 09:56:29.862371 7670 heartbeater.cc:511] Master 127.0.97.60:34989 requested a full tablet report, sending...
I20260924 09:56:29.864445 7347 ts_manager.cc:194] Registered new tserver with Master: 3dc89c3746aa49159412f76b46758c66 (127.0.97.2:35061)
I20260924 09:56:29.865862 7672 heartbeater.cc:347] Connected to a master server at 127.0.97.61:46619
I20260924 09:56:29.865985 7672 heartbeater.cc:464] Registering TS with master...
I20260924 09:56:29.866338 7672 heartbeater.cc:511] Master 127.0.97.61:46619 requested a full tablet report, sending...
I20260924 09:56:29.867501 7163 ts_manager.cc:194] Registered new tserver with Master: 3dc89c3746aa49159412f76b46758c66 (127.0.97.2:35061)
I20260924 09:56:29.871387 7680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:29.834777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:42663 (local address 127.0.97.62:43811)
0924 09:56:29.848210 (+ 13433us) server_negotiation.cc:206] Beginning negotiation
0924 09:56:29.848216 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 09:56:29.848233 (+ 17us) server_negotiation.cc:407] Connection header received
0924 09:56:29.848285 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:56:29.848290 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:56:29.848363 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:56:29.848455 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:56:29.849894 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:29.850851 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:29.853051 (+ 2200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:29.853674 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:29.853785 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:29.858267 (+ 4482us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:56:29.858273 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:56:29.858277 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:56:29.858332 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:56:29.868249 (+ 9917us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:56:29.869069 (+ 820us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:56:29.869075 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:56:29.869079 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:56:29.869254 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:56:29.869696 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:56:29.869701 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:56:29.869705 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:56:29.870217 (+ 512us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:56:29.870423 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0924 09:56:29.870799 (+ 376us) server_negotiation.cc:299] Negotiation successful
0924 09:56:29.871017 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13322,"thread_start_us":13233,"threads_started":1}
I20260924 09:56:29.871675 7681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:29.834770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43811 (local address 127.0.97.2:42663)
0924 09:56:29.847704 (+ 12934us) negotiation.cc:107] Waiting for socket to connect
0924 09:56:29.847722 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 09:56:29.847827 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:56:29.848558 (+ 731us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:56:29.848563 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:56:29.849436 (+ 873us) client_negotiation.cc:189] Negotiated authn=SASL
0924 09:56:29.849717 (+ 281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:29.849727 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:29.851005 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:29.851010 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:29.852907 (+ 1897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:29.852918 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:29.854001 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:29.854007 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:29.854173 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:29.854706 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 09:56:29.854727 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 09:56:29.858062 (+ 3335us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 09:56:29.868406 (+ 10344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 09:56:29.868411 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 09:56:29.868416 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 09:56:29.868934 (+ 518us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 09:56:29.869413 (+ 479us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 09:56:29.869418 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 09:56:29.869422 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 09:56:29.869589 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 09:56:29.870364 (+ 775us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 09:56:29.870369 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 09:56:29.870531 (+ 162us) client_negotiation.cc:769] Sending connection context
0924 09:56:29.870669 (+ 138us) client_negotiation.cc:240] Negotiation successful
0924 09:56:29.870978 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12790,"spinlock_wait_cycles":18688,"thread_start_us":13500,"threads_started":1}
I20260924 09:56:29.872723 7671 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43811
I20260924 09:56:29.872845 7671 heartbeater.cc:464] Registering TS with master...
I20260924 09:56:29.873251 7671 heartbeater.cc:511] Master 127.0.97.62:43811 requested a full tablet report, sending...
I20260924 09:56:29.874155 6559 ts_manager.cc:194] Re-registered known tserver with Master: 3dc89c3746aa49159412f76b46758c66 (127.0.97.2:35061)
I20260924 09:56:29.875206 6559 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:42663
I20260924 09:56:29.950372 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu with pid 6888
I20260924 09:56:29.963969 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
/tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790243735634835-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790243735634835-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790243735634835-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790243735634835-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:33087
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:43811
--webserver_port=34619
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790243735634835-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.97.62:43811,127.0.97.61:46619
--tserver_master_addrs=127.0.97.62:43811,127.0.97.61:46619,127.0.97.60:34989 with env {KRB5CCNAME=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790243735634835-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790243735634835-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790243735634835-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 09:56:30.271912 7683 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 09:56:30.272192 7683 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 09:56:30.272322 7683 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 09:56:30.279425 7683 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 09:56:30.279542 7683 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 09:56:30.279698 7683 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260924 09:56:30.289155 7683 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790243735634835-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790243735634835-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790243735634835-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:33087
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790243735634835-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=34619
--tserver_master_addrs=127.0.97.62:43811,127.0.97.61:46619,127.0.97.60:34989
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790243735634835-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 09:56:30.290510 7683 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 09:56:30.291921 7683 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 09:56:30.300881 7526 heartbeater.cc:503] Master 127.0.97.62:43811 was elected leader, sending a full tablet report...
W20260924 09:56:30.306119 7689 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:56:30.306098 7688 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:56:30.306169 7691 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 09:56:30.306964 7683 server_base.cc:1034] running on GCE node
I20260924 09:56:30.307614 7683 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 09:56:30.308579 7683 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 09:56:30.309764 7683 hybrid_clock.cc:648] HybridClock initialized: now 1790243790309709 us; error 42 us; skew 500 ppm
I20260924 09:56:30.310050 7683 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 09:56:30 dist-test-slave-w8d2 krb5kdc[6515](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243790, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 09:56:30.312932 7683 init.cc:377] Logged in from keytab as oryx/127.0.97.3@KRBTEST.COM (short username oryx)
I20260924 09:56:30.314270 7683 webserver.cc:460] Webserver started at http://127.0.97.3:34619/ using document root <none> and password file <none>
I20260924 09:56:30.314903 7683 fs_manager.cc:359] Metadata directory not provided
I20260924 09:56:30.315024 7683 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 09:56:30.320212 7683 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.002s
I20260924 09:56:30.322777 7698 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 09:56:30.324093 7683 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260924 09:56:30.324250 7683 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790243735634835-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790243735634835-388-0/minicluster-data/ts-2/wal
uuid: "2bdc5f8429c3414e9f322f85ce09088a"
format_stamp: "Formatted at 2026-09-24 09:56:27 on dist-test-slave-w8d2"
server_key: "106fb2e9b3c52fb2c0f57effc7d9722e"
server_key_iv: "5deaea5cae45176116522466c02f07e7"
server_key_version: "encryptionkey@0"
I20260924 09:56:30.325161 7683 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790243735634835-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790243735634835-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790243735634835-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 09:56:30.362102 7683 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 09:56:30.365967 7683 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 09:56:30.366223 7683 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 09:56:30.367164 7683 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 09:56:30.368530 7683 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 09:56:30.368624 7683 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 09:56:30.368721 7683 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 09:56:30.368788 7683 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 09:56:30.397096 7683 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:33087
I20260924 09:56:30.397157 7810 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:33087 every 8 connection(s)
I20260924 09:56:30.398458 7683 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790243735634835-388-0/minicluster-data/ts-2/data/info.pb
I20260924 09:56:30.401161 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu as pid 7683
Sep 24 09:56:30 dist-test-slave-w8d2 krb5kdc[6515](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243790, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
Sep 24 09:56:30 dist-test-slave-w8d2 krb5kdc[6515](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243790, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
Sep 24 09:56:30 dist-test-slave-w8d2 krb5kdc[6515](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243790, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
I20260924 09:56:30.470899 7817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:30.402079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:38731 (local address 127.0.97.62:43811)
0924 09:56:30.412229 (+ 10150us) server_negotiation.cc:206] Beginning negotiation
0924 09:56:30.412236 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 09:56:30.446274 (+ 34038us) server_negotiation.cc:407] Connection header received
0924 09:56:30.446348 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:56:30.446353 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:56:30.446447 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:56:30.446564 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:56:30.449013 (+ 2449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:30.450494 (+ 1481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:30.453055 (+ 2561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:30.454142 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:30.454224 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:30.459935 (+ 5711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:56:30.459942 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:56:30.459947 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:56:30.459983 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:56:30.463718 (+ 3735us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:56:30.468207 (+ 4489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:56:30.468213 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:56:30.468219 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:56:30.468389 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:56:30.468929 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:56:30.468934 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:56:30.468937 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:56:30.469251 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:56:30.469361 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0924 09:56:30.470246 (+ 885us) server_negotiation.cc:299] Negotiation successful
0924 09:56:30.470497 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10017,"thread_start_us":9930,"threads_started":1}
I20260924 09:56:30.473268 7815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:30.401702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43811 (local address 127.0.97.3:38731)
0924 09:56:30.443425 (+ 41723us) negotiation.cc:107] Waiting for socket to connect
0924 09:56:30.443493 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0924 09:56:30.445881 (+ 2388us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:56:30.446929 (+ 1048us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:56:30.446941 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:56:30.447809 (+ 868us) client_negotiation.cc:189] Negotiated authn=SASL
0924 09:56:30.448802 (+ 993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:30.448826 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:30.450677 (+ 1851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:30.450681 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:30.452893 (+ 2212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:30.452908 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:30.454534 (+ 1626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:30.454541 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:30.454816 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:30.456389 (+ 1573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 09:56:30.456417 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0924 09:56:30.459674 (+ 3257us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 09:56:30.463895 (+ 4221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 09:56:30.463906 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 09:56:30.463926 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0924 09:56:30.466570 (+ 2644us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 09:56:30.468536 (+ 1966us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 09:56:30.468540 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 09:56:30.468544 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 09:56:30.468801 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 09:56:30.469453 (+ 652us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 09:56:30.469460 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 09:56:30.469785 (+ 325us) client_negotiation.cc:769] Sending connection context
0924 09:56:30.471172 (+ 1387us) client_negotiation.cc:240] Negotiation successful
0924 09:56:30.471786 (+ 614us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41370,"mutex_wait_us":3,"thread_start_us":41137,"threads_started":1}
I20260924 09:56:30.473645 7829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:30.402344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:39065 (local address 127.0.97.61:46619)
0924 09:56:30.441248 (+ 38904us) server_negotiation.cc:206] Beginning negotiation
0924 09:56:30.441254 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 09:56:30.447384 (+ 6130us) server_negotiation.cc:407] Connection header received
0924 09:56:30.447512 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:56:30.447517 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:56:30.447599 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:56:30.448240 (+ 641us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:56:30.449502 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:30.450568 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:30.457483 (+ 6915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:30.458219 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:30.458334 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:30.464406 (+ 6072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:56:30.464415 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:56:30.464421 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:56:30.464462 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:56:30.467926 (+ 3464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:56:30.468951 (+ 1025us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:56:30.468956 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:56:30.468960 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:56:30.470283 (+ 1323us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:56:30.470676 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:56:30.470681 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:56:30.470684 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:56:30.471516 (+ 832us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:56:30.471656 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0924 09:56:30.472834 (+ 1178us) server_negotiation.cc:299] Negotiation successful
0924 09:56:30.473038 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38765,"thread_start_us":55488,"threads_started":1}
I20260924 09:56:30.474197 7816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:30.401702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:46619 (local address 127.0.97.3:39065)
0924 09:56:30.447271 (+ 45569us) negotiation.cc:107] Waiting for socket to connect
0924 09:56:30.447293 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 09:56:30.447414 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:56:30.448419 (+ 1005us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:56:30.448423 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:56:30.449032 (+ 609us) client_negotiation.cc:189] Negotiated authn=SASL
0924 09:56:30.449338 (+ 306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:30.449349 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:30.450727 (+ 1378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:30.450731 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:30.456082 (+ 5351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:30.456095 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:30.458709 (+ 2614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:30.458716 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:30.458913 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:30.459649 (+ 736us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 09:56:30.459669 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 09:56:30.464204 (+ 4535us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 09:56:30.468132 (+ 3928us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 09:56:30.468137 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 09:56:30.468141 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 09:56:30.468741 (+ 600us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 09:56:30.470410 (+ 1669us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 09:56:30.470414 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 09:56:30.470418 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 09:56:30.470554 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 09:56:30.471713 (+ 1159us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 09:56:30.471717 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 09:56:30.471861 (+ 144us) client_negotiation.cc:769] Sending connection context
0924 09:56:30.471976 (+ 115us) client_negotiation.cc:240] Negotiation successful
0924 09:56:30.472220 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45392,"mutex_wait_us":3773,"spinlock_wait_cycles":51584,"thread_start_us":45060,"threads_started":1}
I20260924 09:56:30.474928 7812 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43811
I20260924 09:56:30.475307 7812 heartbeater.cc:464] Registering TS with master...
I20260924 09:56:30.476266 7827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:30.422246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52646 (local address 127.0.97.62:43811)
0924 09:56:30.447912 (+ 25666us) server_negotiation.cc:206] Beginning negotiation
0924 09:56:30.447918 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 09:56:30.447940 (+ 22us) server_negotiation.cc:407] Connection header received
0924 09:56:30.447989 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:56:30.447994 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:56:30.448056 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:56:30.451313 (+ 3257us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:56:30.451646 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:30.452900 (+ 1254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:30.455591 (+ 2691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:30.456286 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:30.456445 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:30.466170 (+ 9725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:56:30.466177 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:56:30.466181 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:56:30.466218 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:56:30.469920 (+ 3702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:56:30.470734 (+ 814us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:56:30.470739 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:56:30.470742 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:56:30.470891 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:56:30.471341 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:56:30.471346 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:56:30.471349 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:56:30.474440 (+ 3091us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:56:30.475565 (+ 1125us) server_negotiation.cc:1035] Waiting for connection context
0924 09:56:30.475704 (+ 139us) server_negotiation.cc:299] Negotiation successful
0924 09:56:30.475920 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25472,"thread_start_us":25382,"threads_started":1}
I20260924 09:56:30.476264 7812 heartbeater.cc:511] Master 127.0.97.62:43811 requested a full tablet report, sending...
I20260924 09:56:30.476540 7813 heartbeater.cc:347] Connected to a master server at 127.0.97.61:46619
I20260924 09:56:30.476656 7813 heartbeater.cc:464] Registering TS with master...
I20260924 09:56:30.477094 7813 heartbeater.cc:511] Master 127.0.97.61:46619 requested a full tablet report, sending...
I20260924 09:56:30.478529 6559 ts_manager.cc:194] Re-registered known tserver with Master: 2bdc5f8429c3414e9f322f85ce09088a (127.0.97.3:33087)
I20260924 09:56:30.478758 7821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:30.414644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48348 (local address 127.0.97.60:34989)
0924 09:56:30.448569 (+ 33925us) server_negotiation.cc:206] Beginning negotiation
0924 09:56:30.448575 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 09:56:30.448592 (+ 17us) server_negotiation.cc:407] Connection header received
0924 09:56:30.448646 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:56:30.448652 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:56:30.448721 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:56:30.454178 (+ 5457us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:56:30.454338 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:30.455348 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:30.458055 (+ 2707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:30.459303 (+ 1248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:30.460728 (+ 1425us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:30.465311 (+ 4583us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:56:30.465316 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:56:30.465321 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:56:30.465354 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:56:30.472878 (+ 7524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:56:30.473643 (+ 765us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:56:30.473648 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:56:30.473651 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:56:30.473834 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:56:30.474244 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:56:30.474248 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:56:30.474251 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:56:30.474556 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:56:30.474759 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0924 09:56:30.478187 (+ 3428us) server_negotiation.cc:299] Negotiation successful
0924 09:56:30.478418 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33804,"thread_start_us":32031,"threads_started":1}
I20260924 09:56:30.478892 7163 ts_manager.cc:194] Registered new tserver with Master: 2bdc5f8429c3414e9f322f85ce09088a (127.0.97.3:33087)
I20260924 09:56:30.479590 6559 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:38731
I20260924 09:56:30.484030 7818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:30.427389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53222 (local address 127.0.97.61:46619)
0924 09:56:30.458105 (+ 30716us) server_negotiation.cc:206] Beginning negotiation
0924 09:56:30.458112 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 09:56:30.458130 (+ 18us) server_negotiation.cc:407] Connection header received
0924 09:56:30.458193 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:56:30.458197 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:56:30.458280 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:56:30.458610 (+ 330us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:56:30.464199 (+ 5589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:30.465447 (+ 1248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:30.467855 (+ 2408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:30.470358 (+ 2503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:30.471050 (+ 692us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:30.473418 (+ 2368us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:56:30.473424 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:56:30.473429 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:56:30.473465 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:56:30.480880 (+ 7415us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:56:30.482151 (+ 1271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:56:30.482156 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:56:30.482159 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:56:30.482328 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:56:30.482694 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:56:30.482699 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:56:30.482702 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:56:30.482997 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:56:30.483209 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0924 09:56:30.483588 (+ 379us) server_negotiation.cc:299] Negotiation successful
0924 09:56:30.483748 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30569,"thread_start_us":13860,"threads_started":1}
I20260924 09:56:30.485793 7825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:30.403366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:42597 (local address 127.0.97.60:34989)
0924 09:56:30.442077 (+ 38711us) server_negotiation.cc:206] Beginning negotiation
0924 09:56:30.442084 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 09:56:30.449320 (+ 7236us) server_negotiation.cc:407] Connection header received
0924 09:56:30.457080 (+ 7760us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:56:30.457085 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:56:30.457160 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:56:30.457254 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:56:30.460964 (+ 3710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:30.462118 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:30.464126 (+ 2008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:30.464747 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:30.464860 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:30.475821 (+ 10961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:56:30.475827 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:56:30.475831 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:56:30.475859 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:56:30.482140 (+ 6281us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:56:30.482869 (+ 729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:56:30.482873 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:56:30.482877 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:56:30.483000 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:56:30.484470 (+ 1470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:56:30.484476 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:56:30.484479 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:56:30.484788 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:56:30.484917 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0924 09:56:30.485368 (+ 451us) server_negotiation.cc:299] Negotiation successful
0924 09:56:30.485532 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38594,"thread_start_us":44957,"threads_started":1}
I20260924 09:56:30.485972 7820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:30.402872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34989 (local address 127.0.97.3:42597)
0924 09:56:30.449219 (+ 46347us) negotiation.cc:107] Waiting for socket to connect
0924 09:56:30.449237 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 09:56:30.451889 (+ 2652us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:56:30.458272 (+ 6383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:56:30.458277 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:56:30.459706 (+ 1429us) client_negotiation.cc:189] Negotiated authn=SASL
0924 09:56:30.460057 (+ 351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:30.460070 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:30.462391 (+ 2321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:30.462397 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:30.463987 (+ 1590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:30.464004 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:30.465523 (+ 1519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:30.465529 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:30.465696 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:30.466875 (+ 1179us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 09:56:30.466908 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0924 09:56:30.475621 (+ 8713us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 09:56:30.482266 (+ 6645us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 09:56:30.482271 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 09:56:30.482274 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 09:56:30.482724 (+ 450us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 09:56:30.484056 (+ 1332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 09:56:30.484062 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 09:56:30.484066 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 09:56:30.484342 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 09:56:30.484993 (+ 651us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 09:56:30.484998 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 09:56:30.485134 (+ 136us) client_negotiation.cc:769] Sending connection context
0924 09:56:30.485266 (+ 132us) client_negotiation.cc:240] Negotiation successful
0924 09:56:30.485580 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46212,"thread_start_us":46078,"threads_started":1}
I20260924 09:56:30.486786 7811 heartbeater.cc:347] Connected to a master server at 127.0.97.60:34989
I20260924 09:56:30.486913 7811 heartbeater.cc:464] Registering TS with master...
I20260924 09:56:30.487285 7811 heartbeater.cc:511] Master 127.0.97.60:34989 requested a full tablet report, sending...
I20260924 09:56:30.488214 7347 ts_manager.cc:194] Registered new tserver with Master: 2bdc5f8429c3414e9f322f85ce09088a (127.0.97.3:33087)
I20260924 09:56:30.483103 6559 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52646:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 09:56:30.491765 6559 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 09:56:30.520475 7841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:30.511271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:46191 (local address 127.0.0.1:53576)
0924 09:56:30.511767 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0924 09:56:30.511789 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 09:56:30.511900 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:56:30.512682 (+ 782us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:56:30.512687 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:56:30.512714 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 09:56:30.513053 (+ 339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:30.513064 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:30.514976 (+ 1912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:30.514982 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:30.517171 (+ 2189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:30.517187 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:30.519570 (+ 2383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:30.519578 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:30.519811 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:30.519850 (+ 39us) client_negotiation.cc:769] Sending connection context
0924 09:56:30.519945 (+ 95us) client_negotiation.cc:240] Negotiation successful
0924 09:56:30.520096 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":732,"threads_started":1}
I20260924 09:56:30.521575 7842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:30.511578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53576 (local address 127.0.97.1:46191)
0924 09:56:30.512073 (+ 495us) server_negotiation.cc:206] Beginning negotiation
0924 09:56:30.512082 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 09:56:30.512110 (+ 28us) server_negotiation.cc:407] Connection header received
0924 09:56:30.512214 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:56:30.512222 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:56:30.512489 (+ 267us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:56:30.512696 (+ 207us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 09:56:30.513231 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:30.514848 (+ 1617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:30.517351 (+ 2503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:30.518979 (+ 1628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:30.519137 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:30.519397 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0924 09:56:30.521069 (+ 1672us) server_negotiation.cc:299] Negotiation successful
0924 09:56:30.521264 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":232,"threads_started":1}
I20260924 09:56:30.528764 7460 tablet_service.cc:1467] Processing CreateTablet for tablet 2cf374327ddf4f85935bd277f63618b3 (DEFAULT_TABLE table=test-table [id=42db41f6a0484dc7853c007995d78dd9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 09:56:30.530282 7460 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2cf374327ddf4f85935bd277f63618b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 09:56:30.532653 7841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:30.512262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:33087 (local address 127.0.0.1:60330)
0924 09:56:30.520755 (+ 8493us) negotiation.cc:107] Waiting for socket to connect
0924 09:56:30.520784 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0924 09:56:30.520889 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:56:30.526508 (+ 5619us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:56:30.526512 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:56:30.526535 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 09:56:30.526697 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:30.526707 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:30.528613 (+ 1906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:30.528619 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:30.530372 (+ 1753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:30.530386 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:30.531886 (+ 1500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:30.531892 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:30.532065 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:30.532097 (+ 32us) client_negotiation.cc:769] Sending connection context
0924 09:56:30.532178 (+ 81us) client_negotiation.cc:240] Negotiation successful
0924 09:56:30.532347 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8371,"thread_start_us":12356,"threads_started":1}
I20260924 09:56:30.532660 7844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:30.512502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60330 (local address 127.0.97.3:33087)
0924 09:56:30.525972 (+ 13470us) server_negotiation.cc:206] Beginning negotiation
0924 09:56:30.525980 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 09:56:30.526011 (+ 31us) server_negotiation.cc:407] Connection header received
0924 09:56:30.526104 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:56:30.526112 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:56:30.526337 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:56:30.526516 (+ 179us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 09:56:30.526863 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:30.528474 (+ 1611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:30.530497 (+ 2023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:30.531744 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:30.531865 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:30.532059 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0924 09:56:30.532233 (+ 174us) server_negotiation.cc:299] Negotiation successful
0924 09:56:30.532376 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13323,"thread_start_us":13174,"threads_started":1}
I20260924 09:56:30.533180 7839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:30.510622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:35061 (local address 127.0.0.1:56612)
0924 09:56:30.511195 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0924 09:56:30.511217 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 09:56:30.511364 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:56:30.525897 (+ 14533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:56:30.525902 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:56:30.525927 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 09:56:30.526225 (+ 298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:30.526235 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:30.527979 (+ 1744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:30.527984 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:30.529591 (+ 1607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:30.529603 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:30.532308 (+ 2705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:30.532314 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:30.532489 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:30.532519 (+ 30us) client_negotiation.cc:769] Sending connection context
0924 09:56:30.532603 (+ 84us) client_negotiation.cc:240] Negotiation successful
0924 09:56:30.532731 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":295,"threads_started":1}
I20260924 09:56:30.533828 7840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:30.510864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56612 (local address 127.0.97.2:35061)
0924 09:56:30.525216 (+ 14352us) server_negotiation.cc:206] Beginning negotiation
0924 09:56:30.525225 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 09:56:30.525257 (+ 32us) server_negotiation.cc:407] Connection header received
0924 09:56:30.525352 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:56:30.525359 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:56:30.525714 (+ 355us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:56:30.525919 (+ 205us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 09:56:30.526406 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:30.527844 (+ 1438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:30.530542 (+ 2698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:30.531846 (+ 1304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:30.531961 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:30.532140 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0924 09:56:30.533399 (+ 1259us) server_negotiation.cc:299] Negotiation successful
0924 09:56:30.533564 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14111,"thread_start_us":15279,"threads_started":1}
I20260924 09:56:30.536479 7746 tablet_service.cc:1467] Processing CreateTablet for tablet 2cf374327ddf4f85935bd277f63618b3 (DEFAULT_TABLE table=test-table [id=42db41f6a0484dc7853c007995d78dd9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 09:56:30.537040 7605 tablet_service.cc:1467] Processing CreateTablet for tablet 2cf374327ddf4f85935bd277f63618b3 (DEFAULT_TABLE table=test-table [id=42db41f6a0484dc7853c007995d78dd9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 09:56:30.538400 7605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2cf374327ddf4f85935bd277f63618b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 09:56:30.538394 7746 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2cf374327ddf4f85935bd277f63618b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 09:56:30.543011 7845 tablet_bootstrap.cc:492] T 2cf374327ddf4f85935bd277f63618b3 P 87e95d359d14496285f25e40421c70c8: Bootstrap starting.
I20260924 09:56:30.546036 7845 tablet_bootstrap.cc:654] T 2cf374327ddf4f85935bd277f63618b3 P 87e95d359d14496285f25e40421c70c8: Neither blocks nor log segments found. Creating new log.
I20260924 09:56:30.547325 7845 log.cc:824] T 2cf374327ddf4f85935bd277f63618b3 P 87e95d359d14496285f25e40421c70c8: Log is configured to *not* fsync() on all Append() calls
I20260924 09:56:30.549737 7845 tablet_bootstrap.cc:492] T 2cf374327ddf4f85935bd277f63618b3 P 87e95d359d14496285f25e40421c70c8: No bootstrap required, opened a new log
I20260924 09:56:30.549968 7845 ts_tablet_manager.cc:1397] T 2cf374327ddf4f85935bd277f63618b3 P 87e95d359d14496285f25e40421c70c8: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.003s
I20260924 09:56:30.553102 7847 tablet_bootstrap.cc:492] T 2cf374327ddf4f85935bd277f63618b3 P 2bdc5f8429c3414e9f322f85ce09088a: Bootstrap starting.
I20260924 09:56:30.554840 7848 tablet_bootstrap.cc:492] T 2cf374327ddf4f85935bd277f63618b3 P 3dc89c3746aa49159412f76b46758c66: Bootstrap starting.
I20260924 09:56:30.557363 7847 tablet_bootstrap.cc:654] T 2cf374327ddf4f85935bd277f63618b3 P 2bdc5f8429c3414e9f322f85ce09088a: Neither blocks nor log segments found. Creating new log.
I20260924 09:56:30.557382 7848 tablet_bootstrap.cc:654] T 2cf374327ddf4f85935bd277f63618b3 P 3dc89c3746aa49159412f76b46758c66: Neither blocks nor log segments found. Creating new log.
I20260924 09:56:30.558501 7848 log.cc:824] T 2cf374327ddf4f85935bd277f63618b3 P 3dc89c3746aa49159412f76b46758c66: Log is configured to *not* fsync() on all Append() calls
I20260924 09:56:30.558645 7847 log.cc:824] T 2cf374327ddf4f85935bd277f63618b3 P 2bdc5f8429c3414e9f322f85ce09088a: Log is configured to *not* fsync() on all Append() calls
I20260924 09:56:30.558938 7845 raft_consensus.cc:348] T 2cf374327ddf4f85935bd277f63618b3 P 87e95d359d14496285f25e40421c70c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3dc89c3746aa49159412f76b46758c66" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35061 } } peers { permanent_uuid: "87e95d359d14496285f25e40421c70c8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46191 } } peers { permanent_uuid: "2bdc5f8429c3414e9f322f85ce09088a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33087 } }
I20260924 09:56:30.559396 7845 raft_consensus.cc:374] T 2cf374327ddf4f85935bd277f63618b3 P 87e95d359d14496285f25e40421c70c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 09:56:30.559500 7845 raft_consensus.cc:729] T 2cf374327ddf4f85935bd277f63618b3 P 87e95d359d14496285f25e40421c70c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87e95d359d14496285f25e40421c70c8, State: Initialized, Role: FOLLOWER
I20260924 09:56:30.560042 7845 consensus_queue.cc:260] T 2cf374327ddf4f85935bd277f63618b3 P 87e95d359d14496285f25e40421c70c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3dc89c3746aa49159412f76b46758c66" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35061 } } peers { permanent_uuid: "87e95d359d14496285f25e40421c70c8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46191 } } peers { permanent_uuid: "2bdc5f8429c3414e9f322f85ce09088a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33087 } }
I20260924 09:56:30.561733 7845 ts_tablet_manager.cc:1428] T 2cf374327ddf4f85935bd277f63618b3 P 87e95d359d14496285f25e40421c70c8: Time spent starting tablet: real 0.012s user 0.006s sys 0.006s
I20260924 09:56:30.561914 7848 tablet_bootstrap.cc:492] T 2cf374327ddf4f85935bd277f63618b3 P 3dc89c3746aa49159412f76b46758c66: No bootstrap required, opened a new log
I20260924 09:56:30.562162 7848 ts_tablet_manager.cc:1397] T 2cf374327ddf4f85935bd277f63618b3 P 3dc89c3746aa49159412f76b46758c66: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260924 09:56:30.563707 7847 tablet_bootstrap.cc:492] T 2cf374327ddf4f85935bd277f63618b3 P 2bdc5f8429c3414e9f322f85ce09088a: No bootstrap required, opened a new log
I20260924 09:56:30.563957 7847 ts_tablet_manager.cc:1397] T 2cf374327ddf4f85935bd277f63618b3 P 2bdc5f8429c3414e9f322f85ce09088a: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20260924 09:56:30.572227 7847 raft_consensus.cc:348] T 2cf374327ddf4f85935bd277f63618b3 P 2bdc5f8429c3414e9f322f85ce09088a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3dc89c3746aa49159412f76b46758c66" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35061 } } peers { permanent_uuid: "87e95d359d14496285f25e40421c70c8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46191 } } peers { permanent_uuid: "2bdc5f8429c3414e9f322f85ce09088a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33087 } }
I20260924 09:56:30.572564 7847 raft_consensus.cc:374] T 2cf374327ddf4f85935bd277f63618b3 P 2bdc5f8429c3414e9f322f85ce09088a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 09:56:30.572660 7847 raft_consensus.cc:729] T 2cf374327ddf4f85935bd277f63618b3 P 2bdc5f8429c3414e9f322f85ce09088a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2bdc5f8429c3414e9f322f85ce09088a, State: Initialized, Role: FOLLOWER
I20260924 09:56:30.573067 7848 raft_consensus.cc:348] T 2cf374327ddf4f85935bd277f63618b3 P 3dc89c3746aa49159412f76b46758c66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3dc89c3746aa49159412f76b46758c66" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35061 } } peers { permanent_uuid: "87e95d359d14496285f25e40421c70c8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46191 } } peers { permanent_uuid: "2bdc5f8429c3414e9f322f85ce09088a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33087 } }
I20260924 09:56:30.573184 7847 consensus_queue.cc:260] T 2cf374327ddf4f85935bd277f63618b3 P 2bdc5f8429c3414e9f322f85ce09088a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3dc89c3746aa49159412f76b46758c66" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35061 } } peers { permanent_uuid: "87e95d359d14496285f25e40421c70c8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46191 } } peers { permanent_uuid: "2bdc5f8429c3414e9f322f85ce09088a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33087 } }
I20260924 09:56:30.573378 7848 raft_consensus.cc:374] T 2cf374327ddf4f85935bd277f63618b3 P 3dc89c3746aa49159412f76b46758c66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 09:56:30.573464 7848 raft_consensus.cc:729] T 2cf374327ddf4f85935bd277f63618b3 P 3dc89c3746aa49159412f76b46758c66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3dc89c3746aa49159412f76b46758c66, State: Initialized, Role: FOLLOWER
I20260924 09:56:30.573952 7848 consensus_queue.cc:260] T 2cf374327ddf4f85935bd277f63618b3 P 3dc89c3746aa49159412f76b46758c66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3dc89c3746aa49159412f76b46758c66" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35061 } } peers { permanent_uuid: "87e95d359d14496285f25e40421c70c8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46191 } } peers { permanent_uuid: "2bdc5f8429c3414e9f322f85ce09088a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33087 } }
I20260924 09:56:30.574170 7812 heartbeater.cc:503] Master 127.0.97.62:43811 was elected leader, sending a full tablet report...
I20260924 09:56:30.574872 7847 ts_tablet_manager.cc:1428] T 2cf374327ddf4f85935bd277f63618b3 P 2bdc5f8429c3414e9f322f85ce09088a: Time spent starting tablet: real 0.011s user 0.007s sys 0.004s
I20260924 09:56:30.576562 7848 ts_tablet_manager.cc:1428] T 2cf374327ddf4f85935bd277f63618b3 P 3dc89c3746aa49159412f76b46758c66: Time spent starting tablet: real 0.014s user 0.011s sys 0.000s
I20260924 09:56:30.576646 7671 heartbeater.cc:503] Master 127.0.97.62:43811 was elected leader, sending a full tablet report...
W20260924 09:56:30.577561 7673 tablet.cc:2369] T 2cf374327ddf4f85935bd277f63618b3 P 3dc89c3746aa49159412f76b46758c66: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 09:56:30.651273 7814 tablet.cc:2369] T 2cf374327ddf4f85935bd277f63618b3 P 2bdc5f8429c3414e9f322f85ce09088a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 09:56:30.759449 7853 raft_consensus.cc:482] T 2cf374327ddf4f85935bd277f63618b3 P 3dc89c3746aa49159412f76b46758c66 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 09:56:30.759737 7853 raft_consensus.cc:504] T 2cf374327ddf4f85935bd277f63618b3 P 3dc89c3746aa49159412f76b46758c66 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3dc89c3746aa49159412f76b46758c66" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35061 } } peers { permanent_uuid: "87e95d359d14496285f25e40421c70c8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46191 } } peers { permanent_uuid: "2bdc5f8429c3414e9f322f85ce09088a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33087 } }
I20260924 09:56:30.761729 7853 leader_election.cc:288] T 2cf374327ddf4f85935bd277f63618b3 P 3dc89c3746aa49159412f76b46758c66 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 87e95d359d14496285f25e40421c70c8 (127.0.97.1:46191), 2bdc5f8429c3414e9f322f85ce09088a (127.0.97.3:33087)
W20260924 09:56:30.766052 7528 tablet.cc:2369] T 2cf374327ddf4f85935bd277f63618b3 P 87e95d359d14496285f25e40421c70c8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 09:56:30.768306 7842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:30.762172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:39945 (local address 127.0.97.1:46191)
0924 09:56:30.762391 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0924 09:56:30.762398 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 09:56:30.762616 (+ 218us) server_negotiation.cc:407] Connection header received
0924 09:56:30.762832 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:56:30.762837 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:56:30.762921 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:56:30.763009 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 09:56:30.763550 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:30.764429 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:30.766486 (+ 2057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:30.767455 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:30.767554 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:30.767625 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 09:56:30.767940 (+ 315us) server_negotiation.cc:299] Negotiation successful
0924 09:56:30.768061 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 09:56:30.768404 7854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:30.761953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:46191 (local address 127.0.97.2:39945)
0924 09:56:30.762506 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0924 09:56:30.762526 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 09:56:30.762700 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:56:30.763050 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:56:30.763055 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:56:30.763083 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 09:56:30.763419 (+ 336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:30.763430 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:30.764547 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:30.764553 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:30.766328 (+ 1775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:30.766341 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:30.767631 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:30.767637 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:30.767798 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:30.767830 (+ 32us) client_negotiation.cc:769] Sending connection context
0924 09:56:30.767911 (+ 81us) client_negotiation.cc:240] Negotiation successful
0924 09:56:30.768040 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":445,"mutex_wait_us":69,"thread_start_us":255,"threads_started":1}
I20260924 09:56:30.768648 7844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:30.762784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:48469 (local address 127.0.97.3:33087)
0924 09:56:30.762930 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0924 09:56:30.762934 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 09:56:30.763265 (+ 331us) server_negotiation.cc:407] Connection header received
0924 09:56:30.763421 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:56:30.763425 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:56:30.763495 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:56:30.763573 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 09:56:30.764079 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:30.764881 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:30.766925 (+ 2044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:30.767821 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:30.767938 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:30.768030 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0924 09:56:30.768335 (+ 305us) server_negotiation.cc:299] Negotiation successful
0924 09:56:30.768460 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260924 09:56:30.768841 7855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:30.762549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:33087 (local address 127.0.97.2:48469)
0924 09:56:30.763155 (+ 606us) negotiation.cc:107] Waiting for socket to connect
0924 09:56:30.763168 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 09:56:30.763306 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:56:30.763596 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:56:30.763601 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:56:30.763624 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 09:56:30.763940 (+ 316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:30.763951 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:30.765013 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:30.765017 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:30.766731 (+ 1714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:30.766745 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:30.768022 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:30.768028 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:30.768186 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:30.768218 (+ 32us) client_negotiation.cc:769] Sending connection context
0924 09:56:30.768307 (+ 89us) client_negotiation.cc:240] Negotiation successful
0924 09:56:30.768432 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474,"mutex_wait_us":33,"thread_start_us":390,"threads_started":1}
I20260924 09:56:30.769644 7480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cf374327ddf4f85935bd277f63618b3" candidate_uuid: "3dc89c3746aa49159412f76b46758c66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87e95d359d14496285f25e40421c70c8" is_pre_election: true
I20260924 09:56:30.770063 7480 raft_consensus.cc:2454] T 2cf374327ddf4f85935bd277f63618b3 P 87e95d359d14496285f25e40421c70c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3dc89c3746aa49159412f76b46758c66 in term 0.
I20260924 09:56:30.769635 7766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cf374327ddf4f85935bd277f63618b3" candidate_uuid: "3dc89c3746aa49159412f76b46758c66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2bdc5f8429c3414e9f322f85ce09088a" is_pre_election: true
I20260924 09:56:30.770519 7766 raft_consensus.cc:2454] T 2cf374327ddf4f85935bd277f63618b3 P 2bdc5f8429c3414e9f322f85ce09088a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3dc89c3746aa49159412f76b46758c66 in term 0.
I20260924 09:56:30.770645 7561 leader_election.cc:302] T 2cf374327ddf4f85935bd277f63618b3 P 3dc89c3746aa49159412f76b46758c66 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3dc89c3746aa49159412f76b46758c66, 87e95d359d14496285f25e40421c70c8; no voters:
I20260924 09:56:30.771010 7853 raft_consensus.cc:2781] T 2cf374327ddf4f85935bd277f63618b3 P 3dc89c3746aa49159412f76b46758c66 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 09:56:30.771122 7853 raft_consensus.cc:482] T 2cf374327ddf4f85935bd277f63618b3 P 3dc89c3746aa49159412f76b46758c66 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 09:56:30.771214 7853 raft_consensus.cc:3037] T 2cf374327ddf4f85935bd277f63618b3 P 3dc89c3746aa49159412f76b46758c66 [term 0 FOLLOWER]: Advancing to term 1
I20260924 09:56:30.773263 7853 raft_consensus.cc:504] T 2cf374327ddf4f85935bd277f63618b3 P 3dc89c3746aa49159412f76b46758c66 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3dc89c3746aa49159412f76b46758c66" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35061 } } peers { permanent_uuid: "87e95d359d14496285f25e40421c70c8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46191 } } peers { permanent_uuid: "2bdc5f8429c3414e9f322f85ce09088a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33087 } }
I20260924 09:56:30.773775 7853 leader_election.cc:288] T 2cf374327ddf4f85935bd277f63618b3 P 3dc89c3746aa49159412f76b46758c66 [CANDIDATE]: Term 1 election: Requested vote from peers 87e95d359d14496285f25e40421c70c8 (127.0.97.1:46191), 2bdc5f8429c3414e9f322f85ce09088a (127.0.97.3:33087)
I20260924 09:56:30.774170 7480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cf374327ddf4f85935bd277f63618b3" candidate_uuid: "3dc89c3746aa49159412f76b46758c66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87e95d359d14496285f25e40421c70c8"
I20260924 09:56:30.774199 7766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cf374327ddf4f85935bd277f63618b3" candidate_uuid: "3dc89c3746aa49159412f76b46758c66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2bdc5f8429c3414e9f322f85ce09088a"
I20260924 09:56:30.774328 7480 raft_consensus.cc:3037] T 2cf374327ddf4f85935bd277f63618b3 P 87e95d359d14496285f25e40421c70c8 [term 0 FOLLOWER]: Advancing to term 1
I20260924 09:56:30.774358 7766 raft_consensus.cc:3037] T 2cf374327ddf4f85935bd277f63618b3 P 2bdc5f8429c3414e9f322f85ce09088a [term 0 FOLLOWER]: Advancing to term 1
I20260924 09:56:30.776378 7480 raft_consensus.cc:2454] T 2cf374327ddf4f85935bd277f63618b3 P 87e95d359d14496285f25e40421c70c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3dc89c3746aa49159412f76b46758c66 in term 1.
I20260924 09:56:30.776382 7766 raft_consensus.cc:2454] T 2cf374327ddf4f85935bd277f63618b3 P 2bdc5f8429c3414e9f322f85ce09088a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3dc89c3746aa49159412f76b46758c66 in term 1.
I20260924 09:56:30.776701 7561 leader_election.cc:302] T 2cf374327ddf4f85935bd277f63618b3 P 3dc89c3746aa49159412f76b46758c66 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2bdc5f8429c3414e9f322f85ce09088a, 3dc89c3746aa49159412f76b46758c66; no voters:
I20260924 09:56:30.776976 7853 raft_consensus.cc:2781] T 2cf374327ddf4f85935bd277f63618b3 P 3dc89c3746aa49159412f76b46758c66 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 09:56:30.777355 7853 raft_consensus.cc:686] T 2cf374327ddf4f85935bd277f63618b3 P 3dc89c3746aa49159412f76b46758c66 [term 1 LEADER]: Becoming Leader. State: Replica: 3dc89c3746aa49159412f76b46758c66, State: Running, Role: LEADER
I20260924 09:56:30.778033 7853 consensus_queue.cc:237] T 2cf374327ddf4f85935bd277f63618b3 P 3dc89c3746aa49159412f76b46758c66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3dc89c3746aa49159412f76b46758c66" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35061 } } peers { permanent_uuid: "87e95d359d14496285f25e40421c70c8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46191 } } peers { permanent_uuid: "2bdc5f8429c3414e9f322f85ce09088a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33087 } }
I20260924 09:56:30.782538 6560 catalog_manager.cc:5495] T 2cf374327ddf4f85935bd277f63618b3 P 3dc89c3746aa49159412f76b46758c66 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3dc89c3746aa49159412f76b46758c66 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "3dc89c3746aa49159412f76b46758c66" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3dc89c3746aa49159412f76b46758c66" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35061 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "87e95d359d14496285f25e40421c70c8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46191 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2bdc5f8429c3414e9f322f85ce09088a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33087 } health_report { overall_health: UNKNOWN } } }
I20260924 09:56:30.810551 7840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:30.805856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56620 (local address 127.0.97.2:35061)
0924 09:56:30.806014 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0924 09:56:30.806018 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 09:56:30.806034 (+ 16us) server_negotiation.cc:407] Connection header received
0924 09:56:30.806146 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:56:30.806150 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:56:30.806226 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:56:30.806345 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 09:56:30.806711 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:30.807441 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:30.808659 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:30.809102 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:30.809195 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:30.809600 (+ 405us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 09:56:30.810106 (+ 506us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 09:56:30.810208 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0924 09:56:30.810242 (+ 34us) server_negotiation.cc:299] Negotiation successful
0924 09:56:30.810304 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 09:56:30.818156 7480 raft_consensus.cc:1261] T 2cf374327ddf4f85935bd277f63618b3 P 87e95d359d14496285f25e40421c70c8 [term 1 FOLLOWER]: Refusing update from remote peer 3dc89c3746aa49159412f76b46758c66: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 09:56:30.818342 7766 raft_consensus.cc:1261] T 2cf374327ddf4f85935bd277f63618b3 P 2bdc5f8429c3414e9f322f85ce09088a [term 1 FOLLOWER]: Refusing update from remote peer 3dc89c3746aa49159412f76b46758c66: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 09:56:30.819120 7856 consensus_queue.cc:1035] T 2cf374327ddf4f85935bd277f63618b3 P 3dc89c3746aa49159412f76b46758c66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "87e95d359d14496285f25e40421c70c8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 09:56:30.819571 7853 consensus_queue.cc:1035] T 2cf374327ddf4f85935bd277f63618b3 P 3dc89c3746aa49159412f76b46758c66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2bdc5f8429c3414e9f322f85ce09088a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 09:56:30.846774 7860 mvcc.cc:205] Tried to move back new op lower bound from 7332838567181787136 to 7332838567030063104. Current Snapshot: MvccSnapshot[applied={T|T < 7332838567181787136}]
I20260924 09:56:30.851903 6560 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52646:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 09:56:30.852164 6560 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52646:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 09:56:30.858295 6560 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c65adc94d5fe46cb8d1244957f570265: Sending DeleteTablet for 3 replicas of tablet 2cf374327ddf4f85935bd277f63618b3
I20260924 09:56:30.859423 7460 tablet_service.cc:1514] Processing DeleteTablet for tablet 2cf374327ddf4f85935bd277f63618b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 09:56:30 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:53576
I20260924 09:56:30.859460 7605 tablet_service.cc:1514] Processing DeleteTablet for tablet 2cf374327ddf4f85935bd277f63618b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 09:56:30 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:56612
I20260924 09:56:30.859467 7746 tablet_service.cc:1514] Processing DeleteTablet for tablet 2cf374327ddf4f85935bd277f63618b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 09:56:30 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:60330
I20260924 09:56:30.860334 7870 tablet_replica.cc:330] T 2cf374327ddf4f85935bd277f63618b3 P 3dc89c3746aa49159412f76b46758c66: stopping tablet replica
I20260924 09:56:30.860845 7870 raft_consensus.cc:2229] T 2cf374327ddf4f85935bd277f63618b3 P 3dc89c3746aa49159412f76b46758c66 [term 1 LEADER]: Raft consensus shutting down.
I20260924 09:56:30.868737 7870 raft_consensus.cc:2258] T 2cf374327ddf4f85935bd277f63618b3 P 3dc89c3746aa49159412f76b46758c66 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 09:56:30.871810 7870 ts_tablet_manager.cc:1905] T 2cf374327ddf4f85935bd277f63618b3 P 3dc89c3746aa49159412f76b46758c66: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 09:56:30.881444 7869 tablet_replica.cc:330] T 2cf374327ddf4f85935bd277f63618b3 P 2bdc5f8429c3414e9f322f85ce09088a: stopping tablet replica
I20260924 09:56:30.881856 7869 raft_consensus.cc:2229] T 2cf374327ddf4f85935bd277f63618b3 P 2bdc5f8429c3414e9f322f85ce09088a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 09:56:30.882318 7869 raft_consensus.cc:2258] T 2cf374327ddf4f85935bd277f63618b3 P 2bdc5f8429c3414e9f322f85ce09088a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 09:56:30.882992 7871 tablet_replica.cc:330] T 2cf374327ddf4f85935bd277f63618b3 P 87e95d359d14496285f25e40421c70c8: stopping tablet replica
I20260924 09:56:30.883371 7871 raft_consensus.cc:2229] T 2cf374327ddf4f85935bd277f63618b3 P 87e95d359d14496285f25e40421c70c8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 09:56:30.883813 7871 raft_consensus.cc:2258] T 2cf374327ddf4f85935bd277f63618b3 P 87e95d359d14496285f25e40421c70c8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 09:56:30.885155 7869 ts_tablet_manager.cc:1905] T 2cf374327ddf4f85935bd277f63618b3 P 2bdc5f8429c3414e9f322f85ce09088a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 09:56:30.885644 7871 ts_tablet_manager.cc:1905] T 2cf374327ddf4f85935bd277f63618b3 P 87e95d359d14496285f25e40421c70c8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 09:56:30.891916 7871 ts_tablet_manager.cc:1918] T 2cf374327ddf4f85935bd277f63618b3 P 87e95d359d14496285f25e40421c70c8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 09:56:30.892072 7871 log.cc:1196] T 2cf374327ddf4f85935bd277f63618b3 P 87e95d359d14496285f25e40421c70c8: Deleting WAL directory at /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790243735634835-388-0/minicluster-data/ts-0/wal/wals/2cf374327ddf4f85935bd277f63618b3
I20260924 09:56:30.892644 7871 ts_tablet_manager.cc:1939] T 2cf374327ddf4f85935bd277f63618b3 P 87e95d359d14496285f25e40421c70c8: Deleting consensus metadata
I20260924 09:56:30.893049 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu pid 6528
I20260924 09:56:30.894134 6590 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 09:56:30.896345 7869 ts_tablet_manager.cc:1918] T 2cf374327ddf4f85935bd277f63618b3 P 2bdc5f8429c3414e9f322f85ce09088a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 09:56:30.897571 7870 ts_tablet_manager.cc:1918] T 2cf374327ddf4f85935bd277f63618b3 P 3dc89c3746aa49159412f76b46758c66: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 09:56:30.897855 7869 log.cc:1196] T 2cf374327ddf4f85935bd277f63618b3 P 2bdc5f8429c3414e9f322f85ce09088a: Deleting WAL directory at /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790243735634835-388-0/minicluster-data/ts-2/wal/wals/2cf374327ddf4f85935bd277f63618b3
I20260924 09:56:30.898406 7869 ts_tablet_manager.cc:1939] T 2cf374327ddf4f85935bd277f63618b3 P 2bdc5f8429c3414e9f322f85ce09088a: Deleting consensus metadata
I20260924 09:56:30.898609 7870 log.cc:1196] T 2cf374327ddf4f85935bd277f63618b3 P 3dc89c3746aa49159412f76b46758c66: Deleting WAL directory at /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790243735634835-388-0/minicluster-data/ts-1/wal/wals/2cf374327ddf4f85935bd277f63618b3
I20260924 09:56:30.899192 7870 ts_tablet_manager.cc:1939] T 2cf374327ddf4f85935bd277f63618b3 P 3dc89c3746aa49159412f76b46758c66: Deleting consensus metadata
I20260924 09:56:30.909616 7818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:30.861955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53224 (local address 127.0.97.61:46619)
0924 09:56:30.862122 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0924 09:56:30.862128 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 09:56:30.887855 (+ 25727us) server_negotiation.cc:407] Connection header received
0924 09:56:30.888125 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:56:30.888131 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:56:30.888218 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:56:30.888281 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:56:30.890443 (+ 2162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:30.891341 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:30.892907 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:30.893527 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:30.893616 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:30.902523 (+ 8907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:56:30.902530 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:56:30.902534 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:56:30.902577 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:56:30.906372 (+ 3795us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:56:30.907262 (+ 890us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:56:30.907268 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:56:30.907271 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:56:30.907486 (+ 215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:56:30.907964 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:56:30.907969 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:56:30.907972 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:56:30.908309 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:56:30.908529 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0924 09:56:30.909091 (+ 562us) server_negotiation.cc:299] Negotiation successful
0924 09:56:30.909317 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 09:56:30.921901 7825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:30.891824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48354 (local address 127.0.97.60:34989)
0924 09:56:30.892025 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0924 09:56:30.892031 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 09:56:30.909723 (+ 17692us) server_negotiation.cc:407] Connection header received
0924 09:56:30.910152 (+ 429us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:56:30.910159 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:56:30.910244 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:56:30.910325 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:56:30.911551 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:30.912257 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:30.913425 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:30.913828 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:30.913931 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:30.917036 (+ 3105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:56:30.917040 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:56:30.917044 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:56:30.917077 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:56:30.919513 (+ 2436us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:56:30.920182 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:56:30.920187 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:56:30.920190 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:56:30.920317 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:56:30.920641 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:56:30.920645 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:56:30.920647 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:56:30.920899 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:56:30.921113 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0924 09:56:30.921395 (+ 282us) server_negotiation.cc:299] Negotiation successful
0924 09:56:30.921601 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260924 09:56:31.052203 6545 catalog_manager.cc:4841] TS 3dc89c3746aa49159412f76b46758c66 (127.0.97.2:35061): tablet 2cf374327ddf4f85935bd277f63618b3 (table test-table [id=42db41f6a0484dc7853c007995d78dd9]) successfully deleted
W20260924 09:56:31.052456 6545 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.0.97.2:35061, user_credentials={real_user=slave}} blocked reactor thread for 149880us
I20260924 09:56:31.053315 6547 catalog_manager.cc:4841] TS 87e95d359d14496285f25e40421c70c8 (127.0.97.1:46191): tablet 2cf374327ddf4f85935bd277f63618b3 (table test-table [id=42db41f6a0484dc7853c007995d78dd9]) successfully deleted
W20260924 09:56:31.053524 6547 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.0.97.1:46191, user_credentials={real_user=slave}} blocked reactor thread for 153042us
I20260924 09:56:31.053864 6547 catalog_manager.cc:4841] TS 2bdc5f8429c3414e9f322f85ce09088a (127.0.97.3:33087): tablet 2cf374327ddf4f85935bd277f63618b3 (table test-table [id=42db41f6a0484dc7853c007995d78dd9]) successfully deleted
I20260924 09:56:31.055349 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu with pid 6528
I20260924 09:56:31.095204 7825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:31.077818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48362 (local address 127.0.97.60:34989)
0924 09:56:31.077985 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0924 09:56:31.077991 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 09:56:31.080239 (+ 2248us) server_negotiation.cc:407] Connection header received
0924 09:56:31.080400 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:56:31.080407 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:56:31.080496 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:56:31.080592 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:56:31.082798 (+ 2206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:31.083895 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:31.085739 (+ 1844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:31.086369 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:31.086457 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:31.089549 (+ 3092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:56:31.089555 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:56:31.089559 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:56:31.089589 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:56:31.092356 (+ 2767us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:56:31.093081 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:56:31.093087 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:56:31.093090 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:56:31.093221 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:56:31.093583 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:56:31.093588 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:56:31.093591 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:56:31.093929 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:56:31.094162 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0924 09:56:31.094521 (+ 359us) server_negotiation.cc:299] Negotiation successful
0924 09:56:31.094712 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 09:56:31.097271 7818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:31.077442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53232 (local address 127.0.97.61:46619)
0924 09:56:31.077632 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0924 09:56:31.077638 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 09:56:31.081277 (+ 3639us) server_negotiation.cc:407] Connection header received
0924 09:56:31.081690 (+ 413us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:56:31.081696 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:56:31.081772 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:56:31.081838 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:56:31.084076 (+ 2238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:31.085142 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:31.087217 (+ 2075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:31.087803 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:31.087924 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:31.091054 (+ 3130us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:56:31.091061 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:56:31.091065 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:56:31.091103 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:56:31.094185 (+ 3082us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:56:31.094916 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:56:31.094921 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:56:31.094924 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:56:31.095052 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:56:31.095725 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:56:31.095729 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:56:31.095732 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:56:31.096050 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:56:31.096234 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0924 09:56:31.096766 (+ 532us) server_negotiation.cc:299] Negotiation successful
0924 09:56:31.096963 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260924 09:56:31.841343 7881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:31.840151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43811 (local address 127.0.97.1:45557)
0924 09:56:31.841064 (+ 913us) negotiation.cc:107] Waiting for socket to connect
0924 09:56:31.841194 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":785,"thread_start_us":677,"threads_started":1}
W20260924 09:56:31.841677 7526 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:43811 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.97.62:43811: connect: Connection refused (error 111)
I20260924 09:56:31.846975 7882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:31.846266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43811 (local address 127.0.97.2:52135)
0924 09:56:31.846668 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0924 09:56:31.846781 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":209,"threads_started":1}
W20260924 09:56:31.847330 7671 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:43811 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.97.62:43811: connect: Connection refused (error 111)
I20260924 09:56:31.869088 7883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:31.868299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43811 (local address 127.0.97.3:52311)
0924 09:56:31.868716 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0924 09:56:31.868870 (+ 154us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":210,"threads_started":1}
W20260924 09:56:31.869464 7812 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:43811 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.97.62:43811: connect: Connection refused (error 111)
I20260924 09:56:32.358769 7884 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3c8287837bd048888df868ab81b87c63 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c65adc94d5fe46cb8d1244957f570265)
I20260924 09:56:32.359074 7884 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3c8287837bd048888df868ab81b87c63 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c65adc94d5fe46cb8d1244957f570265" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43811 } } peers { permanent_uuid: "b3e4c99bb58342dc915b81f8ea383475" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46619 } attrs { promote: false } } peers { permanent_uuid: "3c8287837bd048888df868ab81b87c63" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34989 } attrs { promote: false } }
I20260924 09:56:32.359441 7885 raft_consensus.cc:482] T 00000000000000000000000000000000 P b3e4c99bb58342dc915b81f8ea383475 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c65adc94d5fe46cb8d1244957f570265)
I20260924 09:56:32.359694 7885 raft_consensus.cc:504] T 00000000000000000000000000000000 P b3e4c99bb58342dc915b81f8ea383475 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c65adc94d5fe46cb8d1244957f570265" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43811 } } peers { permanent_uuid: "b3e4c99bb58342dc915b81f8ea383475" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46619 } attrs { promote: false } } peers { permanent_uuid: "3c8287837bd048888df868ab81b87c63" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34989 } attrs { promote: false } }
I20260924 09:56:32.360793 7884 leader_election.cc:288] T 00000000000000000000000000000000 P 3c8287837bd048888df868ab81b87c63 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c65adc94d5fe46cb8d1244957f570265 (127.0.97.62:43811), b3e4c99bb58342dc915b81f8ea383475 (127.0.97.61:46619)
I20260924 09:56:32.372012 7885 leader_election.cc:288] T 00000000000000000000000000000000 P b3e4c99bb58342dc915b81f8ea383475 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c65adc94d5fe46cb8d1244957f570265 (127.0.97.62:43811), 3c8287837bd048888df868ab81b87c63 (127.0.97.60:34989)
I20260924 09:56:32.374334 7888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:32.361164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43811 (local address 127.0.0.1:53084)
0924 09:56:32.373892 (+ 12728us) negotiation.cc:107] Waiting for socket to connect
0924 09:56:32.374057 (+ 165us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":12338,"thread_start_us":12204,"threads_started":1}
I20260924 09:56:32.374362 7886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:32.360847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43811 (local address 127.0.0.1:53082)
0924 09:56:32.373946 (+ 13099us) negotiation.cc:107] Waiting for socket to connect
0924 09:56:32.374096 (+ 150us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":12987,"thread_start_us":12834,"threads_started":1}
W20260924 09:56:32.374580 7148 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43811: connect: Connection refused (error 111)
W20260924 09:56:32.375156 7332 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43811: connect: Connection refused (error 111)
I20260924 09:56:32.376623 7888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:32.376262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43811 (local address 127.0.0.1:53096)
0924 09:56:32.376406 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0924 09:56:32.376497 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260924 09:56:32.376888 7148 leader_election.cc:334] T 00000000000000000000000000000000 P b3e4c99bb58342dc915b81f8ea383475 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c65adc94d5fe46cb8d1244957f570265 (127.0.97.62:43811): Network error: Client connection negotiation failed: client connection to 127.0.97.62:43811: connect: Connection refused (error 111)
I20260924 09:56:32.377470 7887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:32.376554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43811 (local address 127.0.0.1:53100)
0924 09:56:32.377115 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0924 09:56:32.377276 (+ 161us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
W20260924 09:56:32.377668 7332 leader_election.cc:334] T 00000000000000000000000000000000 P 3c8287837bd048888df868ab81b87c63 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c65adc94d5fe46cb8d1244957f570265 (127.0.97.62:43811): Network error: Client connection negotiation failed: client connection to 127.0.97.62:43811: connect: Connection refused (error 111)
I20260924 09:56:32.381405 7889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:32.361232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34989 (local address 127.0.0.1:48366)
0924 09:56:32.375220 (+ 13988us) negotiation.cc:107] Waiting for socket to connect
0924 09:56:32.375261 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 09:56:32.375376 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:56:32.376000 (+ 624us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:56:32.376005 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:56:32.376033 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 09:56:32.376378 (+ 345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:32.376390 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:32.378105 (+ 1715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:32.378111 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:32.379413 (+ 1302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:32.379423 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:32.380635 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:32.380641 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:32.380839 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:32.380877 (+ 38us) client_negotiation.cc:769] Sending connection context
0924 09:56:32.380968 (+ 91us) client_negotiation.cc:240] Negotiation successful
0924 09:56:32.381095 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13885,"mutex_wait_us":1,"thread_start_us":13779,"threads_started":1}
I20260924 09:56:32.381407 7890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:32.361641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48366 (local address 127.0.97.60:34989)
0924 09:56:32.370817 (+ 9176us) server_negotiation.cc:206] Beginning negotiation
0924 09:56:32.370823 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 09:56:32.375484 (+ 4661us) server_negotiation.cc:407] Connection header received
0924 09:56:32.375555 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:56:32.375559 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:56:32.375630 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:56:32.375728 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 09:56:32.376616 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:32.377988 (+ 1372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:32.379542 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:32.380487 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:32.380572 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:32.380636 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 09:56:32.380980 (+ 344us) server_negotiation.cc:299] Negotiation successful
0924 09:56:32.381064 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9045,"thread_start_us":8898,"threads_started":1}
I20260924 09:56:32.381925 7886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:32.360918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:46619 (local address 127.0.0.1:53238)
0924 09:56:32.374563 (+ 13645us) negotiation.cc:107] Waiting for socket to connect
0924 09:56:32.374580 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 09:56:32.374737 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:56:32.376722 (+ 1985us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:56:32.376727 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:56:32.376749 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 09:56:32.377263 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:32.377281 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:32.378472 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:32.378478 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:32.379979 (+ 1501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:32.379992 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:32.381206 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:32.381212 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:32.381390 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:32.381425 (+ 35us) client_negotiation.cc:769] Sending connection context
0924 09:56:32.381511 (+ 86us) client_negotiation.cc:240] Negotiation successful
0924 09:56:32.381632 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13560,"thread_start_us":14484,"threads_started":1}
I20260924 09:56:32.382014 7357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b3e4c99bb58342dc915b81f8ea383475" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "3c8287837bd048888df868ab81b87c63" is_pre_election: true
I20260924 09:56:32.382318 7891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:32.365192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53238 (local address 127.0.97.61:46619)
0924 09:56:32.376458 (+ 11266us) server_negotiation.cc:206] Beginning negotiation
0924 09:56:32.376463 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 09:56:32.376478 (+ 15us) server_negotiation.cc:407] Connection header received
0924 09:56:32.376524 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:56:32.376528 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:56:32.376588 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:56:32.376883 (+ 295us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 09:56:32.377433 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:32.378338 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:32.380103 (+ 1765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:32.381044 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:32.381139 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:32.381239 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0924 09:56:32.381915 (+ 676us) server_negotiation.cc:299] Negotiation successful
0924 09:56:32.382038 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":22,"server-negotiator.queue_time_us":11149,"thread_start_us":11579,"threads_started":1}
I20260924 09:56:32.382351 7357 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3c8287837bd048888df868ab81b87c63 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3e4c99bb58342dc915b81f8ea383475 in term 1.
I20260924 09:56:32.382831 7147 leader_election.cc:302] T 00000000000000000000000000000000 P b3e4c99bb58342dc915b81f8ea383475 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c8287837bd048888df868ab81b87c63, b3e4c99bb58342dc915b81f8ea383475; no voters: c65adc94d5fe46cb8d1244957f570265
I20260924 09:56:32.382870 7173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3c8287837bd048888df868ab81b87c63" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "b3e4c99bb58342dc915b81f8ea383475" is_pre_election: true
I20260924 09:56:32.383147 7173 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b3e4c99bb58342dc915b81f8ea383475 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c8287837bd048888df868ab81b87c63 in term 1.
I20260924 09:56:32.383332 7885 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b3e4c99bb58342dc915b81f8ea383475 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260924 09:56:32.383462 7885 raft_consensus.cc:482] T 00000000000000000000000000000000 P b3e4c99bb58342dc915b81f8ea383475 [term 1 FOLLOWER]: Starting leader election (detected failure of leader c65adc94d5fe46cb8d1244957f570265)
I20260924 09:56:32.383560 7885 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b3e4c99bb58342dc915b81f8ea383475 [term 1 FOLLOWER]: Advancing to term 2
I20260924 09:56:32.383638 7334 leader_election.cc:302] T 00000000000000000000000000000000 P 3c8287837bd048888df868ab81b87c63 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c8287837bd048888df868ab81b87c63, b3e4c99bb58342dc915b81f8ea383475; no voters: c65adc94d5fe46cb8d1244957f570265
I20260924 09:56:32.383947 7884 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3c8287837bd048888df868ab81b87c63 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260924 09:56:32.384063 7884 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3c8287837bd048888df868ab81b87c63 [term 1 FOLLOWER]: Starting leader election (detected failure of leader c65adc94d5fe46cb8d1244957f570265)
I20260924 09:56:32.384153 7884 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3c8287837bd048888df868ab81b87c63 [term 1 FOLLOWER]: Advancing to term 2
I20260924 09:56:32.385673 7885 raft_consensus.cc:504] T 00000000000000000000000000000000 P b3e4c99bb58342dc915b81f8ea383475 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c65adc94d5fe46cb8d1244957f570265" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43811 } } peers { permanent_uuid: "b3e4c99bb58342dc915b81f8ea383475" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46619 } attrs { promote: false } } peers { permanent_uuid: "3c8287837bd048888df868ab81b87c63" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34989 } attrs { promote: false } }
I20260924 09:56:32.386040 7884 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3c8287837bd048888df868ab81b87c63 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c65adc94d5fe46cb8d1244957f570265" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43811 } } peers { permanent_uuid: "b3e4c99bb58342dc915b81f8ea383475" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46619 } attrs { promote: false } } peers { permanent_uuid: "3c8287837bd048888df868ab81b87c63" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34989 } attrs { promote: false } }
I20260924 09:56:32.386241 7885 leader_election.cc:288] T 00000000000000000000000000000000 P b3e4c99bb58342dc915b81f8ea383475 [CANDIDATE]: Term 2 election: Requested vote from peers c65adc94d5fe46cb8d1244957f570265 (127.0.97.62:43811), 3c8287837bd048888df868ab81b87c63 (127.0.97.60:34989)
I20260924 09:56:32.386674 7884 leader_election.cc:288] T 00000000000000000000000000000000 P 3c8287837bd048888df868ab81b87c63 [CANDIDATE]: Term 2 election: Requested vote from peers c65adc94d5fe46cb8d1244957f570265 (127.0.97.62:43811), b3e4c99bb58342dc915b81f8ea383475 (127.0.97.61:46619)
I20260924 09:56:32.386701 7357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b3e4c99bb58342dc915b81f8ea383475" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "3c8287837bd048888df868ab81b87c63"
I20260924 09:56:32.386991 7357 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 3c8287837bd048888df868ab81b87c63 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate b3e4c99bb58342dc915b81f8ea383475 in current term 2: Already voted for candidate 3c8287837bd048888df868ab81b87c63 in this term.
I20260924 09:56:32.387266 7886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:32.386837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43811 (local address 127.0.0.1:53122)
0924 09:56:32.387037 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0924 09:56:32.387111 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106}
I20260924 09:56:32.387269 7173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3c8287837bd048888df868ab81b87c63" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "b3e4c99bb58342dc915b81f8ea383475"
I20260924 09:56:32.387434 7889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:32.386343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43811 (local address 127.0.0.1:53116)
0924 09:56:32.386705 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0924 09:56:32.386991 (+ 286us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":260}
I20260924 09:56:32.387539 7173 raft_consensus.cc:2379] T 00000000000000000000000000000000 P b3e4c99bb58342dc915b81f8ea383475 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 3c8287837bd048888df868ab81b87c63 in current term 2: Already voted for candidate b3e4c99bb58342dc915b81f8ea383475 in this term.
I20260924 09:56:32.388147 7886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:32.387830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43811 (local address 127.0.0.1:53136)
0924 09:56:32.387951 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0924 09:56:32.388057 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260924 09:56:32.388309 7889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:32.387970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43811 (local address 127.0.0.1:53138)
0924 09:56:32.388101 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0924 09:56:32.388174 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260924 09:56:32.388291 7332 leader_election.cc:334] T 00000000000000000000000000000000 P 3c8287837bd048888df868ab81b87c63 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer c65adc94d5fe46cb8d1244957f570265 (127.0.97.62:43811): Network error: Client connection negotiation failed: client connection to 127.0.97.62:43811: connect: Connection refused (error 111)
I20260924 09:56:32.388372 7332 leader_election.cc:302] T 00000000000000000000000000000000 P 3c8287837bd048888df868ab81b87c63 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c8287837bd048888df868ab81b87c63; no voters: b3e4c99bb58342dc915b81f8ea383475, c65adc94d5fe46cb8d1244957f570265
W20260924 09:56:32.388468 7148 leader_election.cc:334] T 00000000000000000000000000000000 P b3e4c99bb58342dc915b81f8ea383475 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer c65adc94d5fe46cb8d1244957f570265 (127.0.97.62:43811): Network error: Client connection negotiation failed: client connection to 127.0.97.62:43811: connect: Connection refused (error 111)
I20260924 09:56:32.388579 7148 leader_election.cc:302] T 00000000000000000000000000000000 P b3e4c99bb58342dc915b81f8ea383475 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b3e4c99bb58342dc915b81f8ea383475; no voters: 3c8287837bd048888df868ab81b87c63, c65adc94d5fe46cb8d1244957f570265
I20260924 09:56:32.388619 7884 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 3c8287837bd048888df868ab81b87c63 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260924 09:56:32.388818 7885 raft_consensus.cc:2726] T 00000000000000000000000000000000 P b3e4c99bb58342dc915b81f8ea383475 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260924 09:56:32.843076 7894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:32.842280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43811 (local address 127.0.97.1:43599)
0924 09:56:32.842822 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0924 09:56:32.842905 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":325,"threads_started":1}
I20260924 09:56:32.848630 7895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:32.847958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43811 (local address 127.0.97.2:40083)
0924 09:56:32.848373 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0924 09:56:32.848472 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":211,"threads_started":1}
I20260924 09:56:32.870705 7896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:32.870076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43811 (local address 127.0.97.3:48219)
0924 09:56:32.870483 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0924 09:56:32.870561 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":192,"threads_started":1}
I20260924 09:56:33.844518 7897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:33.843828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43811 (local address 127.0.97.1:45279)
0924 09:56:33.844269 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0924 09:56:33.844365 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":223,"threads_started":1}
I20260924 09:56:33.850003 7898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:33.849309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43811 (local address 127.0.97.2:52687)
0924 09:56:33.849756 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0924 09:56:33.849829 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":223,"threads_started":1}
I20260924 09:56:33.872653 7899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:33.871474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43811 (local address 127.0.97.3:43939)
0924 09:56:33.872371 (+ 897us) negotiation.cc:107] Waiting for socket to connect
0924 09:56:33.872460 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":791,"thread_start_us":677,"threads_started":1}
I20260924 09:56:34.017237 7900 raft_consensus.cc:482] T 00000000000000000000000000000000 P b3e4c99bb58342dc915b81f8ea383475 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 09:56:34.017433 7900 raft_consensus.cc:504] T 00000000000000000000000000000000 P b3e4c99bb58342dc915b81f8ea383475 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c65adc94d5fe46cb8d1244957f570265" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43811 } } peers { permanent_uuid: "b3e4c99bb58342dc915b81f8ea383475" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46619 } attrs { promote: false } } peers { permanent_uuid: "3c8287837bd048888df868ab81b87c63" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34989 } attrs { promote: false } }
I20260924 09:56:34.018004 7900 leader_election.cc:288] T 00000000000000000000000000000000 P b3e4c99bb58342dc915b81f8ea383475 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c65adc94d5fe46cb8d1244957f570265 (127.0.97.62:43811), 3c8287837bd048888df868ab81b87c63 (127.0.97.60:34989)
I20260924 09:56:34.018522 7357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b3e4c99bb58342dc915b81f8ea383475" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "3c8287837bd048888df868ab81b87c63" is_pre_election: true
I20260924 09:56:34.018752 7357 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3c8287837bd048888df868ab81b87c63 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3e4c99bb58342dc915b81f8ea383475 in term 2.
I20260924 09:56:34.019011 7901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:34.018235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43811 (local address 127.0.0.1:45000)
0924 09:56:34.018692 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0924 09:56:34.018805 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":247,"threads_started":1}
I20260924 09:56:34.019186 7147 leader_election.cc:302] T 00000000000000000000000000000000 P b3e4c99bb58342dc915b81f8ea383475 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c8287837bd048888df868ab81b87c63, b3e4c99bb58342dc915b81f8ea383475; no voters:
I20260924 09:56:34.019539 7900 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b3e4c99bb58342dc915b81f8ea383475 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260924 09:56:34.019670 7900 raft_consensus.cc:482] T 00000000000000000000000000000000 P b3e4c99bb58342dc915b81f8ea383475 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 09:56:34.019767 7900 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b3e4c99bb58342dc915b81f8ea383475 [term 2 FOLLOWER]: Advancing to term 3
I20260924 09:56:34.020385 7901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:34.020049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43811 (local address 127.0.0.1:45010)
0924 09:56:34.020201 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0924 09:56:34.020267 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260924 09:56:34.020545 7148 leader_election.cc:334] T 00000000000000000000000000000000 P b3e4c99bb58342dc915b81f8ea383475 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c65adc94d5fe46cb8d1244957f570265 (127.0.97.62:43811): Network error: Client connection negotiation failed: client connection to 127.0.97.62:43811: connect: Connection refused (error 111)
I20260924 09:56:34.021914 7900 raft_consensus.cc:504] T 00000000000000000000000000000000 P b3e4c99bb58342dc915b81f8ea383475 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c65adc94d5fe46cb8d1244957f570265" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43811 } } peers { permanent_uuid: "b3e4c99bb58342dc915b81f8ea383475" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46619 } attrs { promote: false } } peers { permanent_uuid: "3c8287837bd048888df868ab81b87c63" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34989 } attrs { promote: false } }
I20260924 09:56:34.022408 7900 leader_election.cc:288] T 00000000000000000000000000000000 P b3e4c99bb58342dc915b81f8ea383475 [CANDIDATE]: Term 3 election: Requested vote from peers c65adc94d5fe46cb8d1244957f570265 (127.0.97.62:43811), 3c8287837bd048888df868ab81b87c63 (127.0.97.60:34989)
I20260924 09:56:34.022825 7901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:34.022493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43811 (local address 127.0.0.1:45018)
0924 09:56:34.022622 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0924 09:56:34.022694 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260924 09:56:34.022873 7357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b3e4c99bb58342dc915b81f8ea383475" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "3c8287837bd048888df868ab81b87c63"
I20260924 09:56:34.023092 7357 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3c8287837bd048888df868ab81b87c63 [term 2 FOLLOWER]: Advancing to term 3
I20260924 09:56:34.023699 7901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:34.023382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43811 (local address 127.0.0.1:45032)
0924 09:56:34.023509 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0924 09:56:34.023579 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260924 09:56:34.024242 7148 leader_election.cc:334] T 00000000000000000000000000000000 P b3e4c99bb58342dc915b81f8ea383475 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer c65adc94d5fe46cb8d1244957f570265 (127.0.97.62:43811): Network error: Client connection negotiation failed: client connection to 127.0.97.62:43811: connect: Connection refused (error 111)
I20260924 09:56:34.025516 7357 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3c8287837bd048888df868ab81b87c63 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3e4c99bb58342dc915b81f8ea383475 in term 3.
I20260924 09:56:34.025874 7147 leader_election.cc:302] T 00000000000000000000000000000000 P b3e4c99bb58342dc915b81f8ea383475 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c8287837bd048888df868ab81b87c63, b3e4c99bb58342dc915b81f8ea383475; no voters: c65adc94d5fe46cb8d1244957f570265
I20260924 09:56:34.026134 7900 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b3e4c99bb58342dc915b81f8ea383475 [term 3 FOLLOWER]: Leader election won for term 3
I20260924 09:56:34.026506 7900 raft_consensus.cc:686] T 00000000000000000000000000000000 P b3e4c99bb58342dc915b81f8ea383475 [term 3 LEADER]: Becoming Leader. State: Replica: b3e4c99bb58342dc915b81f8ea383475, State: Running, Role: LEADER
I20260924 09:56:34.026859 7900 consensus_queue.cc:237] T 00000000000000000000000000000000 P b3e4c99bb58342dc915b81f8ea383475 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c65adc94d5fe46cb8d1244957f570265" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43811 } } peers { permanent_uuid: "b3e4c99bb58342dc915b81f8ea383475" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46619 } attrs { promote: false } } peers { permanent_uuid: "3c8287837bd048888df868ab81b87c63" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34989 } attrs { promote: false } }
I20260924 09:56:34.028288 7903 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3e4c99bb58342dc915b81f8ea383475 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b3e4c99bb58342dc915b81f8ea383475. Latest consensus state: current_term: 3 leader_uuid: "b3e4c99bb58342dc915b81f8ea383475" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c65adc94d5fe46cb8d1244957f570265" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43811 } } peers { permanent_uuid: "b3e4c99bb58342dc915b81f8ea383475" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46619 } attrs { promote: false } } peers { permanent_uuid: "3c8287837bd048888df868ab81b87c63" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34989 } attrs { promote: false } } }
I20260924 09:56:34.028573 7903 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3e4c99bb58342dc915b81f8ea383475 [sys.catalog]: This master's current role is: LEADER
I20260924 09:56:34.029156 7905 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 09:56:34.032423 7905 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 09:56:34.033146 7905 catalog_manager.cc:1259] Loaded cluster ID: 0d84da8a40b44bcdbd4e55f25ff820dc
I20260924 09:56:34.033218 7905 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 09:56:34.034246 7905 catalog_manager.cc:1504] Loading token signing keys...
I20260924 09:56:34.034981 7905 catalog_manager.cc:5879] T 00000000000000000000000000000000 P b3e4c99bb58342dc915b81f8ea383475: Loaded TSK: 0
I20260924 09:56:34.035786 7905 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 09:56:34.109750 7162 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53232:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 09:56:34.116391 7162 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 09:56:34.120368 7901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:34.119972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43811 (local address 127.0.0.1:45054)
0924 09:56:34.120147 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0924 09:56:34.120229 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260924 09:56:34.120398 7357 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3c8287837bd048888df868ab81b87c63 [term 3 FOLLOWER]: Refusing update from remote peer b3e4c99bb58342dc915b81f8ea383475: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260924 09:56:34.121042 7903 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b3e4c99bb58342dc915b81f8ea383475 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c8287837bd048888df868ab81b87c63" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34989 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260924 09:56:34.121223 7901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:34.120908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43811 (local address 127.0.0.1:45068)
0924 09:56:34.121049 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0924 09:56:34.121110 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260924 09:56:34.121384 7148 consensus_peers.cc:487] T 00000000000000000000000000000000 P b3e4c99bb58342dc915b81f8ea383475 -> Peer c65adc94d5fe46cb8d1244957f570265 (127.0.97.62:43811): Couldn't send request to peer c65adc94d5fe46cb8d1244957f570265. Status: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43811: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260924 09:56:34.123342 7907 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c8287837bd048888df868ab81b87c63 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b3e4c99bb58342dc915b81f8ea383475. Latest consensus state: current_term: 3 leader_uuid: "b3e4c99bb58342dc915b81f8ea383475" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c65adc94d5fe46cb8d1244957f570265" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43811 } } peers { permanent_uuid: "b3e4c99bb58342dc915b81f8ea383475" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46619 } attrs { promote: false } } peers { permanent_uuid: "3c8287837bd048888df868ab81b87c63" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34989 } attrs { promote: false } } }
I20260924 09:56:34.123564 7907 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c8287837bd048888df868ab81b87c63 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 09:56:34.124912 7903 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3e4c99bb58342dc915b81f8ea383475 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "b3e4c99bb58342dc915b81f8ea383475" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c65adc94d5fe46cb8d1244957f570265" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43811 } } peers { permanent_uuid: "b3e4c99bb58342dc915b81f8ea383475" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46619 } attrs { promote: false } } peers { permanent_uuid: "3c8287837bd048888df868ab81b87c63" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34989 } attrs { promote: false } } }
I20260924 09:56:34.125207 7903 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3e4c99bb58342dc915b81f8ea383475 [sys.catalog]: This master's current role is: LEADER
I20260924 09:56:34.125407 7907 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c8287837bd048888df868ab81b87c63 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "b3e4c99bb58342dc915b81f8ea383475" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c65adc94d5fe46cb8d1244957f570265" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43811 } } peers { permanent_uuid: "b3e4c99bb58342dc915b81f8ea383475" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46619 } attrs { promote: false } } peers { permanent_uuid: "3c8287837bd048888df868ab81b87c63" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34989 } attrs { promote: false } } }
I20260924 09:56:34.125658 7907 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c8287837bd048888df868ab81b87c63 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 09:56:34.139268 7914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:34.132738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:35061 (local address 127.0.0.1:42514)
0924 09:56:34.133322 (+ 584us) negotiation.cc:107] Waiting for socket to connect
0924 09:56:34.133339 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 09:56:34.133450 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:56:34.134265 (+ 815us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:56:34.134270 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:56:34.134303 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 09:56:34.134573 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:34.134583 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:34.135869 (+ 1286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:34.135873 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:34.137389 (+ 1516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:34.137399 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:34.138505 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:34.138511 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:34.138664 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:34.138698 (+ 34us) client_negotiation.cc:769] Sending connection context
0924 09:56:34.138765 (+ 67us) client_negotiation.cc:240] Negotiation successful
0924 09:56:34.138882 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":478,"spinlock_wait_cycles":4480,"thread_start_us":469,"threads_started":1}
I20260924 09:56:34.139858 7901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:34.132412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:33087 (local address 127.0.0.1:40374)
0924 09:56:34.132761 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0924 09:56:34.132784 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 09:56:34.132893 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:56:34.133996 (+ 1103us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:56:34.134000 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:56:34.134022 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 09:56:34.134260 (+ 238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:34.134271 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:34.135540 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:34.135544 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:34.137619 (+ 2075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:34.137632 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:34.139176 (+ 1544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:34.139182 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:34.139375 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:34.139408 (+ 33us) client_negotiation.cc:769] Sending connection context
0924 09:56:34.139469 (+ 61us) client_negotiation.cc:240] Negotiation successful
0924 09:56:34.139590 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"mutex_wait_us":67}
I20260924 09:56:34.139870 7915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:34.133088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42514 (local address 127.0.97.2:35061)
0924 09:56:34.133986 (+ 898us) server_negotiation.cc:206] Beginning negotiation
0924 09:56:34.133991 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 09:56:34.134006 (+ 15us) server_negotiation.cc:407] Connection header received
0924 09:56:34.134064 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:56:34.134069 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:56:34.134132 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:56:34.134228 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 09:56:34.134734 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:34.135740 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:34.137533 (+ 1793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:34.138341 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:34.139351 (+ 1010us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:34.139450 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0924 09:56:34.139494 (+ 44us) server_negotiation.cc:299] Negotiation successful
0924 09:56:34.139625 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":791,"thread_start_us":225,"threads_started":1}
I20260924 09:56:34.140530 7605 tablet_service.cc:1467] Processing CreateTablet for tablet eefd6cfe3605428cafc1d296f18696a5 (DEFAULT_TABLE table=test-table [id=0a9b968386d94c0ea232d8d9ba58970b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 09:56:34.140707 7913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:34.132526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40374 (local address 127.0.97.3:33087)
0924 09:56:34.132920 (+ 394us) server_negotiation.cc:206] Beginning negotiation
0924 09:56:34.133698 (+ 778us) server_negotiation.cc:399] Waiting for connection header
0924 09:56:34.133730 (+ 32us) server_negotiation.cc:407] Connection header received
0924 09:56:34.133815 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:56:34.133820 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:56:34.133893 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:56:34.133986 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 09:56:34.134417 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:34.135406 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:34.137793 (+ 2387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:34.138983 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:34.140150 (+ 1167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:34.140250 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0924 09:56:34.140296 (+ 46us) server_negotiation.cc:299] Negotiation successful
0924 09:56:34.140414 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":438,"threads_started":1}
I20260924 09:56:34.140972 7605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eefd6cfe3605428cafc1d296f18696a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 09:56:34.141324 7746 tablet_service.cc:1467] Processing CreateTablet for tablet eefd6cfe3605428cafc1d296f18696a5 (DEFAULT_TABLE table=test-table [id=0a9b968386d94c0ea232d8d9ba58970b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 09:56:34.141685 7746 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eefd6cfe3605428cafc1d296f18696a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 09:56:34.145622 7918 tablet_bootstrap.cc:492] T eefd6cfe3605428cafc1d296f18696a5 P 3dc89c3746aa49159412f76b46758c66: Bootstrap starting.
I20260924 09:56:34.145874 7916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:34.133117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:46191 (local address 127.0.0.1:36934)
0924 09:56:34.133734 (+ 617us) negotiation.cc:107] Waiting for socket to connect
0924 09:56:34.133750 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 09:56:34.133848 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:56:34.134746 (+ 898us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:56:34.134750 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:56:34.134772 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 09:56:34.135125 (+ 353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:34.135135 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:34.136300 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:34.136305 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:34.138364 (+ 2059us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:34.138376 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:34.140022 (+ 1646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:34.140028 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:34.145303 (+ 5275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:34.145344 (+ 41us) client_negotiation.cc:769] Sending connection context
0924 09:56:34.145433 (+ 89us) client_negotiation.cc:240] Negotiation successful
0924 09:56:34.145556 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":515,"mutex_wait_us":2,"thread_start_us":383,"threads_started":1}
I20260924 09:56:34.146368 7917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:34.133474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36934 (local address 127.0.97.1:46191)
0924 09:56:34.133978 (+ 504us) server_negotiation.cc:206] Beginning negotiation
0924 09:56:34.133984 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 09:56:34.133999 (+ 15us) server_negotiation.cc:407] Connection header received
0924 09:56:34.134060 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:56:34.134065 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:56:34.134136 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:56:34.134234 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 09:56:34.135315 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:34.136185 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:34.138528 (+ 2343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:34.139692 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:34.139798 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:34.139897 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0924 09:56:34.145825 (+ 5928us) server_negotiation.cc:299] Negotiation successful
0924 09:56:34.145992 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":388,"thread_start_us":249,"threads_started":1}
I20260924 09:56:34.147135 7919 tablet_bootstrap.cc:492] T eefd6cfe3605428cafc1d296f18696a5 P 2bdc5f8429c3414e9f322f85ce09088a: Bootstrap starting.
I20260924 09:56:34.147701 7460 tablet_service.cc:1467] Processing CreateTablet for tablet eefd6cfe3605428cafc1d296f18696a5 (DEFAULT_TABLE table=test-table [id=0a9b968386d94c0ea232d8d9ba58970b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 09:56:34.148038 7460 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eefd6cfe3605428cafc1d296f18696a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 09:56:34.148631 7919 tablet_bootstrap.cc:654] T eefd6cfe3605428cafc1d296f18696a5 P 2bdc5f8429c3414e9f322f85ce09088a: Neither blocks nor log segments found. Creating new log.
I20260924 09:56:34.149847 7918 tablet_bootstrap.cc:654] T eefd6cfe3605428cafc1d296f18696a5 P 3dc89c3746aa49159412f76b46758c66: Neither blocks nor log segments found. Creating new log.
I20260924 09:56:34.150462 7919 tablet_bootstrap.cc:492] T eefd6cfe3605428cafc1d296f18696a5 P 2bdc5f8429c3414e9f322f85ce09088a: No bootstrap required, opened a new log
I20260924 09:56:34.150605 7919 ts_tablet_manager.cc:1397] T eefd6cfe3605428cafc1d296f18696a5 P 2bdc5f8429c3414e9f322f85ce09088a: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260924 09:56:34.151299 7919 raft_consensus.cc:348] T eefd6cfe3605428cafc1d296f18696a5 P 2bdc5f8429c3414e9f322f85ce09088a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bdc5f8429c3414e9f322f85ce09088a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33087 } } peers { permanent_uuid: "3dc89c3746aa49159412f76b46758c66" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35061 } } peers { permanent_uuid: "87e95d359d14496285f25e40421c70c8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46191 } }
I20260924 09:56:34.151543 7919 raft_consensus.cc:374] T eefd6cfe3605428cafc1d296f18696a5 P 2bdc5f8429c3414e9f322f85ce09088a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 09:56:34.151684 7919 raft_consensus.cc:729] T eefd6cfe3605428cafc1d296f18696a5 P 2bdc5f8429c3414e9f322f85ce09088a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2bdc5f8429c3414e9f322f85ce09088a, State: Initialized, Role: FOLLOWER
I20260924 09:56:34.151859 7918 tablet_bootstrap.cc:492] T eefd6cfe3605428cafc1d296f18696a5 P 3dc89c3746aa49159412f76b46758c66: No bootstrap required, opened a new log
I20260924 09:56:34.152029 7918 ts_tablet_manager.cc:1397] T eefd6cfe3605428cafc1d296f18696a5 P 3dc89c3746aa49159412f76b46758c66: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.001s
I20260924 09:56:34.151968 7919 consensus_queue.cc:260] T eefd6cfe3605428cafc1d296f18696a5 P 2bdc5f8429c3414e9f322f85ce09088a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bdc5f8429c3414e9f322f85ce09088a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33087 } } peers { permanent_uuid: "3dc89c3746aa49159412f76b46758c66" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35061 } } peers { permanent_uuid: "87e95d359d14496285f25e40421c70c8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46191 } }
I20260924 09:56:34.152740 7919 ts_tablet_manager.cc:1428] T eefd6cfe3605428cafc1d296f18696a5 P 2bdc5f8429c3414e9f322f85ce09088a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 09:56:34.152755 7918 raft_consensus.cc:348] T eefd6cfe3605428cafc1d296f18696a5 P 3dc89c3746aa49159412f76b46758c66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bdc5f8429c3414e9f322f85ce09088a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33087 } } peers { permanent_uuid: "3dc89c3746aa49159412f76b46758c66" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35061 } } peers { permanent_uuid: "87e95d359d14496285f25e40421c70c8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46191 } }
I20260924 09:56:34.152973 7918 raft_consensus.cc:374] T eefd6cfe3605428cafc1d296f18696a5 P 3dc89c3746aa49159412f76b46758c66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 09:56:34.153103 7918 raft_consensus.cc:729] T eefd6cfe3605428cafc1d296f18696a5 P 3dc89c3746aa49159412f76b46758c66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3dc89c3746aa49159412f76b46758c66, State: Initialized, Role: FOLLOWER
I20260924 09:56:34.153348 7918 consensus_queue.cc:260] T eefd6cfe3605428cafc1d296f18696a5 P 3dc89c3746aa49159412f76b46758c66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bdc5f8429c3414e9f322f85ce09088a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33087 } } peers { permanent_uuid: "3dc89c3746aa49159412f76b46758c66" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35061 } } peers { permanent_uuid: "87e95d359d14496285f25e40421c70c8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46191 } }
I20260924 09:56:34.154829 7899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:34.154357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43811 (local address 127.0.97.3:50343)
0924 09:56:34.154547 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0924 09:56:34.154661 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260924 09:56:34.157963 7813 heartbeater.cc:503] Master 127.0.97.61:46619 was elected leader, sending a full tablet report...
I20260924 09:56:34.158419 7899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:34.157938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43811 (local address 127.0.97.3:48521)
0924 09:56:34.158152 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0924 09:56:34.158246 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
I20260924 09:56:34.163371 7918 ts_tablet_manager.cc:1428] T eefd6cfe3605428cafc1d296f18696a5 P 3dc89c3746aa49159412f76b46758c66: Time spent starting tablet: real 0.011s user 0.002s sys 0.009s
I20260924 09:56:34.163686 7923 tablet_bootstrap.cc:492] T eefd6cfe3605428cafc1d296f18696a5 P 87e95d359d14496285f25e40421c70c8: Bootstrap starting.
I20260924 09:56:34.164955 7898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:34.164208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43811 (local address 127.0.97.2:34769)
0924 09:56:34.164712 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0924 09:56:34.164799 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":399}
I20260924 09:56:34.165869 7898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:34.165489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43811 (local address 127.0.97.2:35431)
0924 09:56:34.165654 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0924 09:56:34.165733 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260924 09:56:34.165920 7923 tablet_bootstrap.cc:654] T eefd6cfe3605428cafc1d296f18696a5 P 87e95d359d14496285f25e40421c70c8: Neither blocks nor log segments found. Creating new log.
I20260924 09:56:34.166373 7672 heartbeater.cc:503] Master 127.0.97.61:46619 was elected leader, sending a full tablet report...
I20260924 09:56:34.167805 7923 tablet_bootstrap.cc:492] T eefd6cfe3605428cafc1d296f18696a5 P 87e95d359d14496285f25e40421c70c8: No bootstrap required, opened a new log
I20260924 09:56:34.167924 7923 ts_tablet_manager.cc:1397] T eefd6cfe3605428cafc1d296f18696a5 P 87e95d359d14496285f25e40421c70c8: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260924 09:56:34.168542 7923 raft_consensus.cc:348] T eefd6cfe3605428cafc1d296f18696a5 P 87e95d359d14496285f25e40421c70c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bdc5f8429c3414e9f322f85ce09088a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33087 } } peers { permanent_uuid: "3dc89c3746aa49159412f76b46758c66" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35061 } } peers { permanent_uuid: "87e95d359d14496285f25e40421c70c8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46191 } }
I20260924 09:56:34.168751 7923 raft_consensus.cc:374] T eefd6cfe3605428cafc1d296f18696a5 P 87e95d359d14496285f25e40421c70c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 09:56:34.168883 7923 raft_consensus.cc:729] T eefd6cfe3605428cafc1d296f18696a5 P 87e95d359d14496285f25e40421c70c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87e95d359d14496285f25e40421c70c8, State: Initialized, Role: FOLLOWER
I20260924 09:56:34.169112 7923 consensus_queue.cc:260] T eefd6cfe3605428cafc1d296f18696a5 P 87e95d359d14496285f25e40421c70c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bdc5f8429c3414e9f322f85ce09088a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33087 } } peers { permanent_uuid: "3dc89c3746aa49159412f76b46758c66" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35061 } } peers { permanent_uuid: "87e95d359d14496285f25e40421c70c8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46191 } }
I20260924 09:56:34.169888 7923 ts_tablet_manager.cc:1428] T eefd6cfe3605428cafc1d296f18696a5 P 87e95d359d14496285f25e40421c70c8: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 09:56:34.170450 7897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:34.170087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43811 (local address 127.0.97.1:45123)
0924 09:56:34.170221 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0924 09:56:34.170302 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260924 09:56:34.171396 7897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:34.171016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43811 (local address 127.0.97.1:48273)
0924 09:56:34.171157 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0924 09:56:34.171257 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260924 09:56:34.171758 7527 heartbeater.cc:503] Master 127.0.97.61:46619 was elected leader, sending a full tablet report...
I20260924 09:56:34.275588 7926 raft_consensus.cc:482] T eefd6cfe3605428cafc1d296f18696a5 P 87e95d359d14496285f25e40421c70c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 09:56:34.275775 7926 raft_consensus.cc:504] T eefd6cfe3605428cafc1d296f18696a5 P 87e95d359d14496285f25e40421c70c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bdc5f8429c3414e9f322f85ce09088a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33087 } } peers { permanent_uuid: "3dc89c3746aa49159412f76b46758c66" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35061 } } peers { permanent_uuid: "87e95d359d14496285f25e40421c70c8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46191 } }
I20260924 09:56:34.277230 7926 leader_election.cc:288] T eefd6cfe3605428cafc1d296f18696a5 P 87e95d359d14496285f25e40421c70c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2bdc5f8429c3414e9f322f85ce09088a (127.0.97.3:33087), 3dc89c3746aa49159412f76b46758c66 (127.0.97.2:35061)
I20260924 09:56:34.282879 7913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:34.277509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:37831 (local address 127.0.97.3:33087)
0924 09:56:34.277690 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0924 09:56:34.277696 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 09:56:34.277910 (+ 214us) server_negotiation.cc:407] Connection header received
0924 09:56:34.277985 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:56:34.277989 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:56:34.278045 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:56:34.278128 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 09:56:34.278540 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:34.279213 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:34.281071 (+ 1858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:34.281979 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:34.282080 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:34.282153 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 09:56:34.282512 (+ 359us) server_negotiation.cc:299] Negotiation successful
0924 09:56:34.282636 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 09:56:34.282876 7897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:34.277425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:33087 (local address 127.0.97.1:37831)
0924 09:56:34.277671 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0924 09:56:34.277690 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 09:56:34.277826 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:56:34.278160 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:56:34.278165 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:56:34.278217 (+ 52us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 09:56:34.278416 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:34.278423 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:34.279367 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:34.279371 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:34.280921 (+ 1550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:34.280934 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:34.282157 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:34.282163 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:34.282338 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:34.282373 (+ 35us) client_negotiation.cc:769] Sending connection context
0924 09:56:34.282461 (+ 88us) client_negotiation.cc:240] Negotiation successful
0924 09:56:34.282569 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"mutex_wait_us":63,"spinlock_wait_cycles":15616}
I20260924 09:56:34.283181 7915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:34.277655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:51921 (local address 127.0.97.2:35061)
0924 09:56:34.277857 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0924 09:56:34.277862 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 09:56:34.278172 (+ 310us) server_negotiation.cc:407] Connection header received
0924 09:56:34.278346 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:56:34.278350 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:56:34.278404 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:56:34.278472 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 09:56:34.278884 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:34.279550 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:34.281228 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:34.282210 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:34.282325 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:34.282418 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0924 09:56:34.282691 (+ 273us) server_negotiation.cc:299] Negotiation successful
0924 09:56:34.282817 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 09:56:34.283175 7927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:34.277455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:35061 (local address 127.0.97.1:51921)
0924 09:56:34.278078 (+ 623us) negotiation.cc:107] Waiting for socket to connect
0924 09:56:34.278091 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 09:56:34.278231 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:56:34.278493 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:56:34.278497 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:56:34.278517 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 09:56:34.278757 (+ 240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:34.278764 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:34.279668 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:34.279673 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:34.281096 (+ 1423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:34.281108 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:34.282384 (+ 1276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:34.282388 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:34.282539 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:34.282571 (+ 32us) client_negotiation.cc:769] Sending connection context
0924 09:56:34.282651 (+ 80us) client_negotiation.cc:240] Negotiation successful
0924 09:56:34.282769 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":539,"spinlock_wait_cycles":8448,"thread_start_us":351,"threads_started":1}
I20260924 09:56:34.283424 7766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eefd6cfe3605428cafc1d296f18696a5" candidate_uuid: "87e95d359d14496285f25e40421c70c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2bdc5f8429c3414e9f322f85ce09088a" is_pre_election: true
I20260924 09:56:34.283653 7766 raft_consensus.cc:2454] T eefd6cfe3605428cafc1d296f18696a5 P 2bdc5f8429c3414e9f322f85ce09088a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87e95d359d14496285f25e40421c70c8 in term 0.
I20260924 09:56:34.284094 7416 leader_election.cc:302] T eefd6cfe3605428cafc1d296f18696a5 P 87e95d359d14496285f25e40421c70c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2bdc5f8429c3414e9f322f85ce09088a, 87e95d359d14496285f25e40421c70c8; no voters:
I20260924 09:56:34.284410 7926 raft_consensus.cc:2781] T eefd6cfe3605428cafc1d296f18696a5 P 87e95d359d14496285f25e40421c70c8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 09:56:34.284504 7926 raft_consensus.cc:482] T eefd6cfe3605428cafc1d296f18696a5 P 87e95d359d14496285f25e40421c70c8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 09:56:34.284371 7625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eefd6cfe3605428cafc1d296f18696a5" candidate_uuid: "87e95d359d14496285f25e40421c70c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3dc89c3746aa49159412f76b46758c66" is_pre_election: true
I20260924 09:56:34.284584 7926 raft_consensus.cc:3037] T eefd6cfe3605428cafc1d296f18696a5 P 87e95d359d14496285f25e40421c70c8 [term 0 FOLLOWER]: Advancing to term 1
I20260924 09:56:34.284719 7625 raft_consensus.cc:2454] T eefd6cfe3605428cafc1d296f18696a5 P 3dc89c3746aa49159412f76b46758c66 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87e95d359d14496285f25e40421c70c8 in term 0.
I20260924 09:56:34.286651 7926 raft_consensus.cc:504] T eefd6cfe3605428cafc1d296f18696a5 P 87e95d359d14496285f25e40421c70c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bdc5f8429c3414e9f322f85ce09088a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33087 } } peers { permanent_uuid: "3dc89c3746aa49159412f76b46758c66" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35061 } } peers { permanent_uuid: "87e95d359d14496285f25e40421c70c8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46191 } }
I20260924 09:56:34.287227 7926 leader_election.cc:288] T eefd6cfe3605428cafc1d296f18696a5 P 87e95d359d14496285f25e40421c70c8 [CANDIDATE]: Term 1 election: Requested vote from peers 2bdc5f8429c3414e9f322f85ce09088a (127.0.97.3:33087), 3dc89c3746aa49159412f76b46758c66 (127.0.97.2:35061)
I20260924 09:56:34.287623 7766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eefd6cfe3605428cafc1d296f18696a5" candidate_uuid: "87e95d359d14496285f25e40421c70c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2bdc5f8429c3414e9f322f85ce09088a"
I20260924 09:56:34.287700 7625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eefd6cfe3605428cafc1d296f18696a5" candidate_uuid: "87e95d359d14496285f25e40421c70c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3dc89c3746aa49159412f76b46758c66"
I20260924 09:56:34.287822 7766 raft_consensus.cc:3037] T eefd6cfe3605428cafc1d296f18696a5 P 2bdc5f8429c3414e9f322f85ce09088a [term 0 FOLLOWER]: Advancing to term 1
I20260924 09:56:34.287868 7625 raft_consensus.cc:3037] T eefd6cfe3605428cafc1d296f18696a5 P 3dc89c3746aa49159412f76b46758c66 [term 0 FOLLOWER]: Advancing to term 1
I20260924 09:56:34.290549 7625 raft_consensus.cc:2454] T eefd6cfe3605428cafc1d296f18696a5 P 3dc89c3746aa49159412f76b46758c66 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87e95d359d14496285f25e40421c70c8 in term 1.
I20260924 09:56:34.290549 7766 raft_consensus.cc:2454] T eefd6cfe3605428cafc1d296f18696a5 P 2bdc5f8429c3414e9f322f85ce09088a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87e95d359d14496285f25e40421c70c8 in term 1.
I20260924 09:56:34.290910 7416 leader_election.cc:302] T eefd6cfe3605428cafc1d296f18696a5 P 87e95d359d14496285f25e40421c70c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2bdc5f8429c3414e9f322f85ce09088a, 87e95d359d14496285f25e40421c70c8; no voters:
I20260924 09:56:34.291160 7926 raft_consensus.cc:2781] T eefd6cfe3605428cafc1d296f18696a5 P 87e95d359d14496285f25e40421c70c8 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 09:56:34.291606 7926 raft_consensus.cc:686] T eefd6cfe3605428cafc1d296f18696a5 P 87e95d359d14496285f25e40421c70c8 [term 1 LEADER]: Becoming Leader. State: Replica: 87e95d359d14496285f25e40421c70c8, State: Running, Role: LEADER
I20260924 09:56:34.292032 7926 consensus_queue.cc:237] T eefd6cfe3605428cafc1d296f18696a5 P 87e95d359d14496285f25e40421c70c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bdc5f8429c3414e9f322f85ce09088a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33087 } } peers { permanent_uuid: "3dc89c3746aa49159412f76b46758c66" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35061 } } peers { permanent_uuid: "87e95d359d14496285f25e40421c70c8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46191 } }
I20260924 09:56:34.294610 7927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:34.294080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43811 (local address 127.0.97.1:48907)
0924 09:56:34.294363 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0924 09:56:34.294472 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145}
I20260924 09:56:34.295529 7927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:34.295128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43811 (local address 127.0.97.1:39501)
0924 09:56:34.295273 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0924 09:56:34.295349 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260924 09:56:34.295420 7162 catalog_manager.cc:5495] T eefd6cfe3605428cafc1d296f18696a5 P 87e95d359d14496285f25e40421c70c8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 87e95d359d14496285f25e40421c70c8 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "87e95d359d14496285f25e40421c70c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bdc5f8429c3414e9f322f85ce09088a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33087 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3dc89c3746aa49159412f76b46758c66" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35061 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "87e95d359d14496285f25e40421c70c8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46191 } health_report { overall_health: HEALTHY } } }
I20260924 09:56:34.329550 7917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:34.324451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36936 (local address 127.0.97.1:46191)
0924 09:56:34.324647 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0924 09:56:34.324653 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 09:56:34.324672 (+ 19us) server_negotiation.cc:407] Connection header received
0924 09:56:34.324836 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:56:34.324840 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:56:34.324911 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:56:34.325034 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 09:56:34.325361 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:34.326062 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:34.327538 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:34.327929 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:34.328037 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:34.328450 (+ 413us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 09:56:34.328982 (+ 532us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 09:56:34.329120 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0924 09:56:34.329203 (+ 83us) server_negotiation.cc:299] Negotiation successful
0924 09:56:34.329281 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 09:56:34.333235 7766 raft_consensus.cc:1261] T eefd6cfe3605428cafc1d296f18696a5 P 2bdc5f8429c3414e9f322f85ce09088a [term 1 FOLLOWER]: Refusing update from remote peer 87e95d359d14496285f25e40421c70c8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 09:56:34.333467 7625 raft_consensus.cc:1261] T eefd6cfe3605428cafc1d296f18696a5 P 3dc89c3746aa49159412f76b46758c66 [term 1 FOLLOWER]: Refusing update from remote peer 87e95d359d14496285f25e40421c70c8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 09:56:34.333868 7928 consensus_queue.cc:1035] T eefd6cfe3605428cafc1d296f18696a5 P 87e95d359d14496285f25e40421c70c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2bdc5f8429c3414e9f322f85ce09088a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 09:56:34.334178 7926 consensus_queue.cc:1035] T eefd6cfe3605428cafc1d296f18696a5 P 87e95d359d14496285f25e40421c70c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3dc89c3746aa49159412f76b46758c66" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35061 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 09:56:34.336318 7899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:34.335823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43811 (local address 127.0.97.3:35203)
0924 09:56:34.336060 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0924 09:56:34.336149 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127}
I20260924 09:56:34.337157 7898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:34.336793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43811 (local address 127.0.97.2:39203)
0924 09:56:34.336923 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0924 09:56:34.337011 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260924 09:56:34.339098 7931 mvcc.cc:205] Tried to move back new op lower bound from 7332838581583093760 to 7332838581423218688. Current Snapshot: MvccSnapshot[applied={T|T < 7332838581583093760}]
I20260924 09:56:34.340395 7899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:34.339925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43811 (local address 127.0.97.3:58653)
0924 09:56:34.340166 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0924 09:56:34.340245 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139}
I20260924 09:56:34.340842 7927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:34.340403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43811 (local address 127.0.97.1:40869)
0924 09:56:34.340602 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0924 09:56:34.340705 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260924 09:56:34.350641 7927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:34.350205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43811 (local address 127.0.97.1:51607)
0924 09:56:34.350373 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0924 09:56:34.350467 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260924 09:56:34.352169 7163 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53232:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 09:56:34.352419 7163 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53232:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 09:56:34.352722 7898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:34.352261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43811 (local address 127.0.97.2:46605)
0924 09:56:34.352462 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0924 09:56:34.352543 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
I20260924 09:56:34.359556 7163 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b3e4c99bb58342dc915b81f8ea383475: Sending DeleteTablet for 3 replicas of tablet eefd6cfe3605428cafc1d296f18696a5
I20260924 09:56:34.360507 7746 tablet_service.cc:1514] Processing DeleteTablet for tablet eefd6cfe3605428cafc1d296f18696a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 09:56:34 UTC) from {username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:40374
I20260924 09:56:34.361171 7941 tablet_replica.cc:330] T eefd6cfe3605428cafc1d296f18696a5 P 2bdc5f8429c3414e9f322f85ce09088a: stopping tablet replica
I20260924 09:56:34.361430 7941 raft_consensus.cc:2229] T eefd6cfe3605428cafc1d296f18696a5 P 2bdc5f8429c3414e9f322f85ce09088a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 09:56:34.361794 7941 raft_consensus.cc:2258] T eefd6cfe3605428cafc1d296f18696a5 P 2bdc5f8429c3414e9f322f85ce09088a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 09:56:34.361836 7460 tablet_service.cc:1514] Processing DeleteTablet for tablet eefd6cfe3605428cafc1d296f18696a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 09:56:34 UTC) from {username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:36934
I20260924 09:56:34.362207 7605 tablet_service.cc:1514] Processing DeleteTablet for tablet eefd6cfe3605428cafc1d296f18696a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 09:56:34 UTC) from {username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:42514
I20260924 09:56:34.363013 7941 ts_tablet_manager.cc:1905] T eefd6cfe3605428cafc1d296f18696a5 P 2bdc5f8429c3414e9f322f85ce09088a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 09:56:34.364784 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu pid 7327
I20260924 09:56:34.369506 7941 ts_tablet_manager.cc:1918] T eefd6cfe3605428cafc1d296f18696a5 P 2bdc5f8429c3414e9f322f85ce09088a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 09:56:34.369729 7941 log.cc:1196] T eefd6cfe3605428cafc1d296f18696a5 P 2bdc5f8429c3414e9f322f85ce09088a: Deleting WAL directory at /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790243735634835-388-0/minicluster-data/ts-2/wal/wals/eefd6cfe3605428cafc1d296f18696a5
I20260924 09:56:34.370213 7941 ts_tablet_manager.cc:1939] T eefd6cfe3605428cafc1d296f18696a5 P 2bdc5f8429c3414e9f322f85ce09088a: Deleting consensus metadata
I20260924 09:56:34.371402 7150 catalog_manager.cc:4841] TS 2bdc5f8429c3414e9f322f85ce09088a (127.0.97.3:33087): tablet eefd6cfe3605428cafc1d296f18696a5 (table test-table [id=0a9b968386d94c0ea232d8d9ba58970b]) successfully deleted
I20260924 09:56:34.375283 7943 tablet_replica.cc:330] T eefd6cfe3605428cafc1d296f18696a5 P 3dc89c3746aa49159412f76b46758c66: stopping tablet replica
I20260924 09:56:34.375479 7943 raft_consensus.cc:2229] T eefd6cfe3605428cafc1d296f18696a5 P 3dc89c3746aa49159412f76b46758c66 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 09:56:34.375754 7943 raft_consensus.cc:2258] T eefd6cfe3605428cafc1d296f18696a5 P 3dc89c3746aa49159412f76b46758c66 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 09:56:34.376792 7943 ts_tablet_manager.cc:1905] T eefd6cfe3605428cafc1d296f18696a5 P 3dc89c3746aa49159412f76b46758c66: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 09:56:34.378733 7942 tablet_replica.cc:330] T eefd6cfe3605428cafc1d296f18696a5 P 87e95d359d14496285f25e40421c70c8: stopping tablet replica
I20260924 09:56:34.378914 7942 raft_consensus.cc:2229] T eefd6cfe3605428cafc1d296f18696a5 P 87e95d359d14496285f25e40421c70c8 [term 1 LEADER]: Raft consensus shutting down.
I20260924 09:56:34.380370 7942 raft_consensus.cc:2258] T eefd6cfe3605428cafc1d296f18696a5 P 87e95d359d14496285f25e40421c70c8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 09:56:34.381397 7942 ts_tablet_manager.cc:1905] T eefd6cfe3605428cafc1d296f18696a5 P 87e95d359d14496285f25e40421c70c8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 09:56:34.383316 7943 ts_tablet_manager.cc:1918] T eefd6cfe3605428cafc1d296f18696a5 P 3dc89c3746aa49159412f76b46758c66: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 09:56:34.383482 7943 log.cc:1196] T eefd6cfe3605428cafc1d296f18696a5 P 3dc89c3746aa49159412f76b46758c66: Deleting WAL directory at /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790243735634835-388-0/minicluster-data/ts-1/wal/wals/eefd6cfe3605428cafc1d296f18696a5
I20260924 09:56:34.383948 7943 ts_tablet_manager.cc:1939] T eefd6cfe3605428cafc1d296f18696a5 P 3dc89c3746aa49159412f76b46758c66: Deleting consensus metadata
I20260924 09:56:34.384948 7148 catalog_manager.cc:4841] TS 3dc89c3746aa49159412f76b46758c66 (127.0.97.2:35061): tablet eefd6cfe3605428cafc1d296f18696a5 (table test-table [id=0a9b968386d94c0ea232d8d9ba58970b]) successfully deleted
I20260924 09:56:34.387717 7942 ts_tablet_manager.cc:1918] T eefd6cfe3605428cafc1d296f18696a5 P 87e95d359d14496285f25e40421c70c8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 09:56:34.387887 7942 log.cc:1196] T eefd6cfe3605428cafc1d296f18696a5 P 87e95d359d14496285f25e40421c70c8: Deleting WAL directory at /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790243735634835-388-0/minicluster-data/ts-0/wal/wals/eefd6cfe3605428cafc1d296f18696a5
I20260924 09:56:34.388260 7942 ts_tablet_manager.cc:1939] T eefd6cfe3605428cafc1d296f18696a5 P 87e95d359d14496285f25e40421c70c8: Deleting consensus metadata
I20260924 09:56:34.389077 7150 catalog_manager.cc:4841] TS 87e95d359d14496285f25e40421c70c8 (127.0.97.1:46191): tablet eefd6cfe3605428cafc1d296f18696a5 (table test-table [id=0a9b968386d94c0ea232d8d9ba58970b]) successfully deleted
I20260924 09:56:34.395506 7917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:34.366718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36952 (local address 127.0.97.1:46191)
0924 09:56:34.366907 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0924 09:56:34.366914 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 09:56:34.380759 (+ 13845us) server_negotiation.cc:407] Connection header received
0924 09:56:34.381181 (+ 422us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:56:34.381187 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:56:34.381279 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:56:34.381348 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:56:34.383133 (+ 1785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:34.384411 (+ 1278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:34.386151 (+ 1740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:34.386795 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:34.386918 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:34.390045 (+ 3127us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:56:34.390056 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:56:34.390071 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:56:34.390102 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:56:34.392942 (+ 2840us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:56:34.393638 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:56:34.393645 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:56:34.393648 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:56:34.393805 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:56:34.394172 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:56:34.394176 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:56:34.394179 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:56:34.394498 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:56:34.394745 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0924 09:56:34.395013 (+ 268us) server_negotiation.cc:299] Negotiation successful
0924 09:56:34.395159 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 09:56:34.396334 7520 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 09:56:34.501180 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu with pid 7327
I20260924 09:56:34.515775 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu pid 7537
I20260924 09:56:34.528168 7915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:34.516451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42526 (local address 127.0.97.2:35061)
0924 09:56:34.516616 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0924 09:56:34.516623 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 09:56:34.516695 (+ 72us) server_negotiation.cc:407] Connection header received
0924 09:56:34.516867 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:56:34.516871 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:56:34.516931 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:56:34.517021 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:56:34.518220 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:34.518912 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:34.520267 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:34.520649 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:34.520753 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:34.523010 (+ 2257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:56:34.523025 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:56:34.523041 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:56:34.523071 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:56:34.525626 (+ 2555us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:56:34.526236 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:56:34.526245 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:56:34.526247 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:56:34.526367 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:56:34.526755 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:56:34.526760 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:56:34.526763 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:56:34.527106 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:56:34.527385 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0924 09:56:34.527659 (+ 274us) server_negotiation.cc:299] Negotiation successful
0924 09:56:34.527854 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 09:56:34.528792 7665 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 09:56:34.628962 7916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:34.628532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43811 (local address 127.0.0.1:45070)
0924 09:56:34.628704 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0924 09:56:34.628791 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260924 09:56:34.630155 7916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:34.629816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43811 (local address 127.0.0.1:45076)
0924 09:56:34.629959 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0924 09:56:34.630032 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260924 09:56:34.641405 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu with pid 7537
I20260924 09:56:34.658305 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu pid 7683
I20260924 09:56:34.669672 7913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:34.658993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40390 (local address 127.0.97.3:33087)
0924 09:56:34.659161 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0924 09:56:34.659166 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 09:56:34.659185 (+ 19us) server_negotiation.cc:407] Connection header received
0924 09:56:34.659379 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:56:34.659383 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:56:34.659443 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:56:34.659519 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:56:34.660665 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:34.661396 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:34.662499 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:34.662863 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:34.662947 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:34.665172 (+ 2225us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:56:34.665186 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:56:34.665202 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:56:34.665239 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:56:34.667625 (+ 2386us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:56:34.668153 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:56:34.668159 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:56:34.668161 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:56:34.668278 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:56:34.668545 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:56:34.668547 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:56:34.668549 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:56:34.668838 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:56:34.669064 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0924 09:56:34.669243 (+ 179us) server_negotiation.cc:299] Negotiation successful
0924 09:56:34.669447 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 09:56:34.670258 7806 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 09:56:34.762598 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu with pid 7683
I20260924 09:56:34.775671 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu pid 7021
I20260924 09:56:34.776602 7193 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 09:56:34.894598 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu with pid 7021
W20260924 09:56:34.910526 7331 connection.cc:535] server connection from 127.0.0.1:48366 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260924 09:56:34.910800 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu pid 7122
I20260924 09:56:34.911736 7377 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 09:56:35.022432 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu with pid 7122
2026-09-24T09:56:35Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (10708 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790243735634835-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 09:56:35 dist-test-slave-w8d2 krb5kdc[7953](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 09:56:35 dist-test-slave-w8d2 krb5kdc[7953](info): set up 2 sockets
Sep 24 09:56:35 dist-test-slave-w8d2 krb5kdc[7953](info): commencing operation
krb5kdc: starting...
W20260924 09:56:36.015842 388 mini_kdc.cc:121] Time spent starting KDC: real 0.955s user 0.000s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790243735634835-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790243735634835-388-0/krb5kdc/test-user.keytab.
Sep 24 09:56:36 dist-test-slave-w8d2 krb5kdc[7953](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243796, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T09:56:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T09:56:36Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790243735634835-388-0/krb5kdc/oryx_127.0.97.62.keytab.
Entry for principal oryx/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790243735634835-388-0/krb5kdc/oryx_127.0.97.62.keytab.
I20260924 09:56:36.199546 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
/tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790243735634835-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790243735634835-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790243735634835-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790243735634835-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38371
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790243735634835-388-0/krb5kdc/oryx_127.0.97.62.keytab
--principal=oryx/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:38371
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790243735634835-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790243735634835-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790243735634835-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 09:56:36.497803 7966 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 09:56:36.498064 7966 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 09:56:36.498188 7966 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 09:56:36.505052 7966 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 09:56:36.505146 7966 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260924 09:56:36.505205 7966 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 09:56:36.505287 7966 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 09:56:36.505330 7966 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 09:56:36.505393 7966 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 09:56:36.514719 7966 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790243735634835-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790243735634835-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:38371
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790243735634835-388-0/krb5kdc/oryx_127.0.97.62.keytab
--principal=oryx/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38371
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790243735634835-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790243735634835-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 09:56:36.515897 7966 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 09:56:36.517133 7966 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 09:56:36.527809 7971 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:56:36.527745 7972 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:56:36.527761 7974 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 09:56:36.528652 7966 server_base.cc:1034] running on GCE node
I20260924 09:56:36.529289 7966 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 09:56:36.530525 7966 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 09:56:36.531723 7966 hybrid_clock.cc:648] HybridClock initialized: now 1790243796531666 us; error 33 us; skew 500 ppm
I20260924 09:56:36.532012 7966 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 09:56:36 dist-test-slave-w8d2 krb5kdc[7953](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243796, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 09:56:36.534863 7966 init.cc:377] Logged in from keytab as oryx/127.0.97.62@KRBTEST.COM (short username oryx)
I20260924 09:56:36.536120 7966 webserver.cc:460] Webserver started at http://127.0.97.62:36243/ using document root <none> and password file <none>
I20260924 09:56:36.536758 7966 fs_manager.cc:359] Metadata directory not provided
I20260924 09:56:36.536846 7966 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 09:56:36.537130 7966 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 09:56:36.539857 7966 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790243735634835-388-0/minicluster-data/master-0/data/instance:
uuid: "e3f28e3acf1e44e59f3ed21ee9c9bd10"
format_stamp: "Formatted at 2026-09-24 09:56:36 on dist-test-slave-w8d2"
server_key: "8cba619e9b56df478a3951c972d920bc"
server_key_iv: "632ca23844270cad98bcbee364746f18"
server_key_version: "encryptionkey@0"
I20260924 09:56:36.540575 7966 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790243735634835-388-0/minicluster-data/master-0/wal/instance:
uuid: "e3f28e3acf1e44e59f3ed21ee9c9bd10"
format_stamp: "Formatted at 2026-09-24 09:56:36 on dist-test-slave-w8d2"
server_key: "8cba619e9b56df478a3951c972d920bc"
server_key_iv: "632ca23844270cad98bcbee364746f18"
server_key_version: "encryptionkey@0"
I20260924 09:56:36.545190 7966 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 09:56:36.548287 7981 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 09:56:36.549435 7966 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260924 09:56:36.549572 7966 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790243735634835-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790243735634835-388-0/minicluster-data/master-0/wal
uuid: "e3f28e3acf1e44e59f3ed21ee9c9bd10"
format_stamp: "Formatted at 2026-09-24 09:56:36 on dist-test-slave-w8d2"
server_key: "8cba619e9b56df478a3951c972d920bc"
server_key_iv: "632ca23844270cad98bcbee364746f18"
server_key_version: "encryptionkey@0"
I20260924 09:56:36.549747 7966 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790243735634835-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790243735634835-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790243735634835-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 09:56:36.574620 7966 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 09:56:36.578596 7966 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 09:56:36.578826 7966 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 09:56:36.597023 7966 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:38371
I20260924 09:56:36.597040 8042 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:38371 every 8 connection(s)
I20260924 09:56:36.598227 7966 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790243735634835-388-0/minicluster-data/master-0/data/info.pb
I20260924 09:56:36.601634 8043 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 09:56:36.606555 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu as pid 7966
I20260924 09:56:36.606747 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790243735634835-388-0/minicluster-data/master-0/wal/instance
I20260924 09:56:36.607360 388 external_mini_cluster.cc:1428] Setting key a6904bb4b17cf56da0137be358f30a96
I20260924 09:56:36.611372 8043 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e3f28e3acf1e44e59f3ed21ee9c9bd10: Bootstrap starting.
I20260924 09:56:36.614593 8043 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e3f28e3acf1e44e59f3ed21ee9c9bd10: Neither blocks nor log segments found. Creating new log.
I20260924 09:56:36.615669 8043 log.cc:824] T 00000000000000000000000000000000 P e3f28e3acf1e44e59f3ed21ee9c9bd10: Log is configured to *not* fsync() on all Append() calls
I20260924 09:56:36.618086 8043 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e3f28e3acf1e44e59f3ed21ee9c9bd10: No bootstrap required, opened a new log
Sep 24 09:56:36 dist-test-slave-w8d2 krb5kdc[7953](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243796, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260924 09:56:36.625487 8043 raft_consensus.cc:348] T 00000000000000000000000000000000 P e3f28e3acf1e44e59f3ed21ee9c9bd10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3f28e3acf1e44e59f3ed21ee9c9bd10" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38371 } }
I20260924 09:56:36.625797 8043 raft_consensus.cc:374] T 00000000000000000000000000000000 P e3f28e3acf1e44e59f3ed21ee9c9bd10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 09:56:36.625890 8043 raft_consensus.cc:729] T 00000000000000000000000000000000 P e3f28e3acf1e44e59f3ed21ee9c9bd10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3f28e3acf1e44e59f3ed21ee9c9bd10, State: Initialized, Role: FOLLOWER
I20260924 09:56:36.626520 8043 consensus_queue.cc:260] T 00000000000000000000000000000000 P e3f28e3acf1e44e59f3ed21ee9c9bd10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3f28e3acf1e44e59f3ed21ee9c9bd10" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38371 } }
I20260924 09:56:36.626780 8043 raft_consensus.cc:388] T 00000000000000000000000000000000 P e3f28e3acf1e44e59f3ed21ee9c9bd10 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 09:56:36.626930 8043 raft_consensus.cc:482] T 00000000000000000000000000000000 P e3f28e3acf1e44e59f3ed21ee9c9bd10 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 09:56:36.627104 8043 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e3f28e3acf1e44e59f3ed21ee9c9bd10 [term 0 FOLLOWER]: Advancing to term 1
I20260924 09:56:36.627893 8046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:36.609167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36678 (local address 127.0.97.62:38371)
0924 09:56:36.609948 (+ 781us) server_negotiation.cc:206] Beginning negotiation
0924 09:56:36.609958 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 09:56:36.609995 (+ 37us) server_negotiation.cc:407] Connection header received
0924 09:56:36.611591 (+ 1596us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:56:36.611612 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:56:36.611968 (+ 356us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:56:36.612256 (+ 288us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:56:36.613788 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:36.615068 (+ 1280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:36.616618 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:36.617234 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:36.617392 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:36.621078 (+ 3686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:56:36.621088 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:56:36.621101 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:56:36.621135 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:56:36.623949 (+ 2814us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:56:36.624539 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:56:36.624545 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:56:36.624551 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:56:36.624699 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:56:36.625021 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:56:36.625026 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:56:36.625028 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:56:36.625471 (+ 443us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:56:36.625732 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0924 09:56:36.626121 (+ 389us) server_negotiation.cc:299] Negotiation successful
0924 09:56:36.626510 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":584,"thread_start_us":363,"threads_started":1}
I20260924 09:56:36.629505 8043 raft_consensus.cc:504] T 00000000000000000000000000000000 P e3f28e3acf1e44e59f3ed21ee9c9bd10 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3f28e3acf1e44e59f3ed21ee9c9bd10" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38371 } }
I20260924 09:56:36.629951 8043 leader_election.cc:302] T 00000000000000000000000000000000 P e3f28e3acf1e44e59f3ed21ee9c9bd10 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e3f28e3acf1e44e59f3ed21ee9c9bd10; no voters:
I20260924 09:56:36.630489 8043 leader_election.cc:288] T 00000000000000000000000000000000 P e3f28e3acf1e44e59f3ed21ee9c9bd10 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 09:56:36.630564 8048 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e3f28e3acf1e44e59f3ed21ee9c9bd10 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 09:56:36.631158 8048 raft_consensus.cc:686] T 00000000000000000000000000000000 P e3f28e3acf1e44e59f3ed21ee9c9bd10 [term 1 LEADER]: Becoming Leader. State: Replica: e3f28e3acf1e44e59f3ed21ee9c9bd10, State: Running, Role: LEADER
I20260924 09:56:36.631685 8048 consensus_queue.cc:237] T 00000000000000000000000000000000 P e3f28e3acf1e44e59f3ed21ee9c9bd10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3f28e3acf1e44e59f3ed21ee9c9bd10" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38371 } }
I20260924 09:56:36.632226 8043 sys_catalog.cc:564] T 00000000000000000000000000000000 P e3f28e3acf1e44e59f3ed21ee9c9bd10 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 09:56:36.634042 8050 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3f28e3acf1e44e59f3ed21ee9c9bd10 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e3f28e3acf1e44e59f3ed21ee9c9bd10. Latest consensus state: current_term: 1 leader_uuid: "e3f28e3acf1e44e59f3ed21ee9c9bd10" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3f28e3acf1e44e59f3ed21ee9c9bd10" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38371 } } }
I20260924 09:56:36.634356 8050 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3f28e3acf1e44e59f3ed21ee9c9bd10 [sys.catalog]: This master's current role is: LEADER
I20260924 09:56:36.634330 8049 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3f28e3acf1e44e59f3ed21ee9c9bd10 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e3f28e3acf1e44e59f3ed21ee9c9bd10" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3f28e3acf1e44e59f3ed21ee9c9bd10" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38371 } } }
I20260924 09:56:36.636541 8049 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3f28e3acf1e44e59f3ed21ee9c9bd10 [sys.catalog]: This master's current role is: LEADER
I20260924 09:56:36.636603 8054 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 09:56:36.641214 8054 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 09:56:36.649075 8054 catalog_manager.cc:1347] Generated new cluster ID: aa4e44fa0acb444299db520acb96ea0a
I20260924 09:56:36.649168 8054 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 09:56:36.667624 8054 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 09:56:36.668862 8054 catalog_manager.cc:1504] Loading token signing keys...
I20260924 09:56:36.675369 8054 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e3f28e3acf1e44e59f3ed21ee9c9bd10: Generated new TSK 0
I20260924 09:56:36.676106 8054 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790243735634835-388-0/krb5kdc/oryx_127.0.97.1.keytab.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790243735634835-388-0/krb5kdc/oryx_127.0.97.1.keytab.
I20260924 09:56:36.720953 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
/tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790243735634835-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790243735634835-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790243735634835-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790243735634835-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38371
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790243735634835-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790243735634835-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790243735634835-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790243735634835-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 09:56:37.025226 8069 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 09:56:37.025496 8069 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 09:56:37.025630 8069 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 09:56:37.032701 8069 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 09:56:37.032812 8069 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 09:56:37.032939 8069 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 09:56:37.033061 8069 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260924 09:56:37.042593 8069 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790243735634835-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790243735634835-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790243735634835-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790243735634835-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:38371
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790243735634835-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 09:56:37.043802 8069 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 09:56:37.045042 8069 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 09:56:37.056730 8075 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:56:37.056756 8074 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:56:37.056958 8077 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 09:56:37.057659 8069 server_base.cc:1034] running on GCE node
I20260924 09:56:37.058198 8069 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 09:56:37.059049 8069 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 09:56:37.060252 8069 hybrid_clock.cc:648] HybridClock initialized: now 1790243797060183 us; error 41 us; skew 500 ppm
I20260924 09:56:37.060544 8069 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 09:56:37 dist-test-slave-w8d2 krb5kdc[7953](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243797, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 09:56:37.063434 8069 init.cc:377] Logged in from keytab as oryx/127.0.97.1@KRBTEST.COM (short username oryx)
I20260924 09:56:37.064706 8069 webserver.cc:460] Webserver started at http://127.0.97.1:36909/ using document root <none> and password file <none>
I20260924 09:56:37.065287 8069 fs_manager.cc:359] Metadata directory not provided
I20260924 09:56:37.065373 8069 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 09:56:37.065619 8069 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 09:56:37.068284 8069 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790243735634835-388-0/minicluster-data/ts-0/data/instance:
uuid: "3e6632b5e12e420283aca499e3c94c42"
format_stamp: "Formatted at 2026-09-24 09:56:37 on dist-test-slave-w8d2"
server_key: "a425ec10b81435d278101eec186658ac"
server_key_iv: "724fab5cec24b1c80c506374fa287742"
server_key_version: "encryptionkey@0"
I20260924 09:56:37.068957 8069 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790243735634835-388-0/minicluster-data/ts-0/wal/instance:
uuid: "3e6632b5e12e420283aca499e3c94c42"
format_stamp: "Formatted at 2026-09-24 09:56:37 on dist-test-slave-w8d2"
server_key: "a425ec10b81435d278101eec186658ac"
server_key_iv: "724fab5cec24b1c80c506374fa287742"
server_key_version: "encryptionkey@0"
I20260924 09:56:37.073529 8069 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 09:56:37.076524 8084 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 09:56:37.077739 8069 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 09:56:37.077860 8069 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790243735634835-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790243735634835-388-0/minicluster-data/ts-0/wal
uuid: "3e6632b5e12e420283aca499e3c94c42"
format_stamp: "Formatted at 2026-09-24 09:56:37 on dist-test-slave-w8d2"
server_key: "a425ec10b81435d278101eec186658ac"
server_key_iv: "724fab5cec24b1c80c506374fa287742"
server_key_version: "encryptionkey@0"
I20260924 09:56:37.078038 8069 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790243735634835-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790243735634835-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790243735634835-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 09:56:37.093621 8069 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 09:56:37.097158 8069 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 09:56:37.097370 8069 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 09:56:37.099701 8069 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 09:56:37.099778 8069 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 09:56:37.099896 8069 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 09:56:37.099951 8069 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 09:56:37 dist-test-slave-w8d2 krb5kdc[7953](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243797, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260924 09:56:37.121279 8046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:37.100256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:55919 (local address 127.0.97.62:38371)
0924 09:56:37.100459 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0924 09:56:37.100465 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 09:56:37.101704 (+ 1239us) server_negotiation.cc:407] Connection header received
0924 09:56:37.103971 (+ 2267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:56:37.103978 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:56:37.104062 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:56:37.104132 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:56:37.106264 (+ 2132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:37.107197 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:37.109920 (+ 2723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:37.110439 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:37.110556 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:37.114679 (+ 4123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:56:37.114685 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:56:37.114689 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:56:37.114721 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:56:37.117838 (+ 3117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:56:37.118679 (+ 841us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:56:37.118684 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:56:37.118688 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:56:37.118841 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:56:37.119377 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:56:37.119382 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:56:37.119385 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:56:37.119732 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:56:37.119867 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0924 09:56:37.120740 (+ 873us) server_negotiation.cc:299] Negotiation successful
0924 09:56:37.120955 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260924 09:56:37.122606 8093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:37.100735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38371 (local address 127.0.97.1:55919)
0924 09:56:37.101527 (+ 792us) negotiation.cc:107] Waiting for socket to connect
0924 09:56:37.101569 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 09:56:37.103654 (+ 2085us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:56:37.104617 (+ 963us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:56:37.104631 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:56:37.105285 (+ 654us) client_negotiation.cc:189] Negotiated authn=SASL
0924 09:56:37.106095 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:37.106122 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:37.107367 (+ 1245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:37.107373 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:37.109746 (+ 2373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:37.109762 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:37.110658 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:37.110665 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:37.110879 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:37.111657 (+ 778us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 09:56:37.111693 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0924 09:56:37.114479 (+ 2786us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 09:56:37.117998 (+ 3519us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 09:56:37.118008 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 09:56:37.118024 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 09:56:37.118558 (+ 534us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 09:56:37.118963 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 09:56:37.118967 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 09:56:37.118971 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 09:56:37.119262 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 09:56:37.119879 (+ 617us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 09:56:37.119893 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 09:56:37.120321 (+ 428us) client_negotiation.cc:769] Sending connection context
0924 09:56:37.120641 (+ 320us) client_negotiation.cc:240] Negotiation successful
0924 09:56:37.121168 (+ 527us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":530,"thread_start_us":315,"threads_started":1}
I20260924 09:56:37.133436 8069 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:40363
I20260924 09:56:37.133497 8203 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:40363 every 8 connection(s)
I20260924 09:56:37.134634 8069 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790243735634835-388-0/minicluster-data/ts-0/data/info.pb
I20260924 09:56:37.136480 8204 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38371
I20260924 09:56:37.136679 8204 heartbeater.cc:464] Registering TS with master...
I20260924 09:56:37.137333 8204 heartbeater.cc:511] Master 127.0.97.62:38371 requested a full tablet report, sending...
I20260924 09:56:37.138024 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu as pid 8069
I20260924 09:56:37.138223 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790243735634835-388-0/minicluster-data/ts-0/wal/instance
I20260924 09:56:37.138826 388 external_mini_cluster.cc:1428] Setting key 8e0fc63a923e1ff8523a34c6324c7286
I20260924 09:56:37.139595 7998 ts_manager.cc:194] Registered new tserver with Master: 3e6632b5e12e420283aca499e3c94c42 (127.0.97.1:40363)
I20260924 09:56:37.141572 7998 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:55919
I20260924 09:56:37.144591 8046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:37.129148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:49495 (local address 127.0.97.62:38371)
0924 09:56:37.129307 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0924 09:56:37.129312 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 09:56:37.131789 (+ 2477us) server_negotiation.cc:407] Connection header received
0924 09:56:37.131970 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:56:37.131975 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:56:37.132042 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:56:37.132121 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:56:37.133503 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:37.134256 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:37.135936 (+ 1680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:37.136590 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:37.136710 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:37.138847 (+ 2137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:56:37.138853 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:56:37.138857 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:56:37.138897 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:56:37.141966 (+ 3069us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:56:37.142689 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:56:37.142694 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:56:37.142698 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:56:37.142844 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:56:37.143208 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:56:37.143212 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:56:37.143215 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:56:37.143545 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:56:37.143677 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0924 09:56:37.144076 (+ 399us) server_negotiation.cc:299] Negotiation successful
0924 09:56:37.144279 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 09:56:37.144654 8191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:37.128913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38371 (local address 127.0.97.1:49495)
0924 09:56:37.131683 (+ 2770us) negotiation.cc:107] Waiting for socket to connect
0924 09:56:37.131702 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 09:56:37.131863 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:56:37.132281 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:56:37.132286 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:56:37.132890 (+ 604us) client_negotiation.cc:189] Negotiated authn=SASL
0924 09:56:37.133311 (+ 421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:37.133326 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:37.134396 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:37.134400 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:37.135821 (+ 1421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:37.135832 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:37.136937 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:37.136941 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:37.137115 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:37.137741 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 09:56:37.137764 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 09:56:37.138673 (+ 909us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 09:56:37.142113 (+ 3440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 09:56:37.142118 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 09:56:37.142122 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 09:56:37.142580 (+ 458us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 09:56:37.142949 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 09:56:37.142953 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 09:56:37.142956 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 09:56:37.143092 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 09:56:37.143671 (+ 579us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 09:56:37.143676 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 09:56:37.143843 (+ 167us) client_negotiation.cc:769] Sending connection context
0924 09:56:37.143974 (+ 131us) client_negotiation.cc:240] Negotiation successful
0924 09:56:37.144238 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2625,"mutex_wait_us":1,"thread_start_us":2499,"threads_started":1}
WARNING: no policy specified for oryx/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790243735634835-388-0/krb5kdc/oryx_127.0.97.2.keytab.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790243735634835-388-0/krb5kdc/oryx_127.0.97.2.keytab.
I20260924 09:56:37.179256 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
/tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790243735634835-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790243735634835-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790243735634835-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790243735634835-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38371
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790243735634835-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790243735634835-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790243735634835-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790243735634835-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 09:56:37.480202 8208 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 09:56:37.480472 8208 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 09:56:37.480605 8208 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 09:56:37.487676 8208 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 09:56:37.487787 8208 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 09:56:37.487903 8208 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 09:56:37.488008 8208 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260924 09:56:37.497654 8208 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790243735634835-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790243735634835-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790243735634835-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790243735634835-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:38371
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790243735634835-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 09:56:37.498888 8208 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 09:56:37.500183 8208 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 09:56:37.512324 8213 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:56:37.512364 8216 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:56:37.512341 8214 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 09:56:37.513116 8208 server_base.cc:1034] running on GCE node
I20260924 09:56:37.513717 8208 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 09:56:37.514596 8208 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 09:56:37.515837 8208 hybrid_clock.cc:648] HybridClock initialized: now 1790243797515762 us; error 43 us; skew 500 ppm
I20260924 09:56:37.516144 8208 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 09:56:37 dist-test-slave-w8d2 krb5kdc[7953](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243797, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 09:56:37.519168 8208 init.cc:377] Logged in from keytab as oryx/127.0.97.2@KRBTEST.COM (short username oryx)
I20260924 09:56:37.520696 8208 webserver.cc:460] Webserver started at http://127.0.97.2:41479/ using document root <none> and password file <none>
I20260924 09:56:37.521361 8208 fs_manager.cc:359] Metadata directory not provided
I20260924 09:56:37.521452 8208 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 09:56:37.521747 8208 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 09:56:37.524613 8208 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790243735634835-388-0/minicluster-data/ts-1/data/instance:
uuid: "43f12639d53d4d19b96ec67379f6fa1f"
format_stamp: "Formatted at 2026-09-24 09:56:37 on dist-test-slave-w8d2"
server_key: "71af12ff50b31f9a75287fb79428928a"
server_key_iv: "a2d91369d672e499190a77ab6426a70a"
server_key_version: "encryptionkey@0"
I20260924 09:56:37.525328 8208 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790243735634835-388-0/minicluster-data/ts-1/wal/instance:
uuid: "43f12639d53d4d19b96ec67379f6fa1f"
format_stamp: "Formatted at 2026-09-24 09:56:37 on dist-test-slave-w8d2"
server_key: "71af12ff50b31f9a75287fb79428928a"
server_key_iv: "a2d91369d672e499190a77ab6426a70a"
server_key_version: "encryptionkey@0"
I20260924 09:56:37.530043 8208 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260924 09:56:37.533183 8223 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 09:56:37.534436 8208 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 09:56:37.534570 8208 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790243735634835-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790243735634835-388-0/minicluster-data/ts-1/wal
uuid: "43f12639d53d4d19b96ec67379f6fa1f"
format_stamp: "Formatted at 2026-09-24 09:56:37 on dist-test-slave-w8d2"
server_key: "71af12ff50b31f9a75287fb79428928a"
server_key_iv: "a2d91369d672e499190a77ab6426a70a"
server_key_version: "encryptionkey@0"
I20260924 09:56:37.534747 8208 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790243735634835-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790243735634835-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790243735634835-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 09:56:37.547155 8208 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 09:56:37.551188 8208 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 09:56:37.551446 8208 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 09:56:37.554014 8208 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 09:56:37.554108 8208 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 09:56:37.554234 8208 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 09:56:37.554314 8208 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 09:56:37 dist-test-slave-w8d2 krb5kdc[7953](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243797, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260924 09:56:37.573952 8046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:37.554446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:36889 (local address 127.0.97.62:38371)
0924 09:56:37.554622 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0924 09:56:37.554628 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 09:56:37.555949 (+ 1321us) server_negotiation.cc:407] Connection header received
0924 09:56:37.557912 (+ 1963us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:56:37.557917 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:56:37.558002 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:56:37.558093 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:56:37.560039 (+ 1946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:37.561010 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:37.563439 (+ 2429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:37.564120 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:37.564259 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:37.567921 (+ 3662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:56:37.567929 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:56:37.567933 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:56:37.567974 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:56:37.570850 (+ 2876us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:56:37.571734 (+ 884us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:56:37.571741 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:56:37.571745 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:56:37.571892 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:56:37.572271 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:56:37.572276 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:56:37.572279 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:56:37.572624 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:56:37.572778 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0924 09:56:37.573330 (+ 552us) server_negotiation.cc:299] Negotiation successful
0924 09:56:37.573585 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 09:56:37.574692 8232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:37.555079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38371 (local address 127.0.97.2:36889)
0924 09:56:37.555810 (+ 731us) negotiation.cc:107] Waiting for socket to connect
0924 09:56:37.555847 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 09:56:37.557628 (+ 1781us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:56:37.558316 (+ 688us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:56:37.558328 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:56:37.559035 (+ 707us) client_negotiation.cc:189] Negotiated authn=SASL
0924 09:56:37.559881 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:37.559907 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:37.561166 (+ 1259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:37.561172 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:37.563178 (+ 2006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:37.563189 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:37.564358 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:37.564366 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:37.564578 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:37.565198 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 09:56:37.565222 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 09:56:37.567659 (+ 2437us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 09:56:37.570981 (+ 3322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 09:56:37.570991 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 09:56:37.571009 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0924 09:56:37.571595 (+ 586us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 09:56:37.572007 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 09:56:37.572011 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 09:56:37.572013 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 09:56:37.572178 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 09:56:37.572728 (+ 550us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 09:56:37.572734 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 09:56:37.573012 (+ 278us) client_negotiation.cc:769] Sending connection context
0924 09:56:37.573214 (+ 202us) client_negotiation.cc:240] Negotiation successful
0924 09:56:37.573552 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":309,"threads_started":1}
I20260924 09:56:37.593152 8281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:37.580436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38371 (local address 127.0.97.2:34541)
0924 09:56:37.581092 (+ 656us) negotiation.cc:107] Waiting for socket to connect
0924 09:56:37.581116 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0924 09:56:37.581263 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:56:37.581585 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:56:37.581594 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:56:37.582057 (+ 463us) client_negotiation.cc:189] Negotiated authn=SASL
0924 09:56:37.582314 (+ 257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:37.582324 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:37.583277 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:37.583283 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:37.584719 (+ 1436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:37.584733 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:37.585671 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:37.585677 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:37.585901 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:37.586491 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 09:56:37.586507 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 09:56:37.587166 (+ 659us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 09:56:37.590837 (+ 3671us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 09:56:37.590841 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 09:56:37.590844 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 09:56:37.591305 (+ 461us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 09:56:37.591688 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 09:56:37.591692 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 09:56:37.591694 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 09:56:37.591800 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 09:56:37.592366 (+ 566us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 09:56:37.592369 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 09:56:37.592508 (+ 139us) client_negotiation.cc:769] Sending connection context
0924 09:56:37.592609 (+ 101us) client_negotiation.cc:240] Negotiation successful
0924 09:56:37.592844 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":336,"threads_started":1}
I20260924 09:56:37.593289 8046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:37.580338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:34541 (local address 127.0.97.62:38371)
0924 09:56:37.580490 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0924 09:56:37.580495 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 09:56:37.581205 (+ 710us) server_negotiation.cc:407] Connection header received
0924 09:56:37.581385 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:56:37.581390 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:56:37.581463 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:56:37.581555 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:56:37.582436 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:37.583128 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:37.584861 (+ 1733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:37.585490 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:37.585618 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:37.587373 (+ 1755us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:56:37.587380 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:56:37.587386 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:56:37.587422 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:56:37.590721 (+ 3299us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:56:37.591421 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:56:37.591427 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:56:37.591431 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:56:37.591588 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:56:37.591881 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:56:37.591886 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:56:37.591890 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:56:37.592269 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:56:37.592412 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0924 09:56:37.592717 (+ 305us) server_negotiation.cc:299] Negotiation successful
0924 09:56:37.592943 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 09:56:37.604442 8208 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:41195
I20260924 09:56:37.604460 8342 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:41195 every 8 connection(s)
I20260924 09:56:37.605778 8208 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790243735634835-388-0/minicluster-data/ts-1/data/info.pb
I20260924 09:56:37.607652 8343 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38371
I20260924 09:56:37.607843 8343 heartbeater.cc:464] Registering TS with master...
I20260924 09:56:37.608398 8343 heartbeater.cc:511] Master 127.0.97.62:38371 requested a full tablet report, sending...
I20260924 09:56:37.609527 7998 ts_manager.cc:194] Registered new tserver with Master: 43f12639d53d4d19b96ec67379f6fa1f (127.0.97.2:41195)
I20260924 09:56:37.610304 7998 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:36889
I20260924 09:56:37.615322 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu as pid 8208
I20260924 09:56:37.615468 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790243735634835-388-0/minicluster-data/ts-1/wal/instance
I20260924 09:56:37.615832 388 external_mini_cluster.cc:1428] Setting key 5b8538d57a9935b05f02559dbe02b8a0
WARNING: no policy specified for oryx/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790243735634835-388-0/krb5kdc/oryx_127.0.97.3.keytab.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790243735634835-388-0/krb5kdc/oryx_127.0.97.3.keytab.
I20260924 09:56:37.654033 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
/tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790243735634835-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790243735634835-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790243735634835-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790243735634835-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38371
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790243735634835-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790243735634835-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790243735634835-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790243735634835-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 09:56:37.952805 8347 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 09:56:37.953137 8347 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 09:56:37.953284 8347 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 09:56:37.960333 8347 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 09:56:37.960430 8347 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 09:56:37.960559 8347 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 09:56:37.960666 8347 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260924 09:56:37.970100 8347 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790243735634835-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790243735634835-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790243735634835-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790243735634835-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:38371
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790243735634835-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 09:56:37.971343 8347 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 09:56:37.972615 8347 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 09:56:37.984618 8352 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:56:37.984578 8355 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:56:37.984597 8353 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 09:56:37.985443 8347 server_base.cc:1034] running on GCE node
I20260924 09:56:37.985970 8347 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 09:56:37.986783 8347 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 09:56:37.987973 8347 hybrid_clock.cc:648] HybridClock initialized: now 1790243797987916 us; error 30 us; skew 500 ppm
I20260924 09:56:37.988245 8347 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 09:56:37 dist-test-slave-w8d2 krb5kdc[7953](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243797, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 09:56:37.991147 8347 init.cc:377] Logged in from keytab as oryx/127.0.97.3@KRBTEST.COM (short username oryx)
I20260924 09:56:37.992488 8347 webserver.cc:460] Webserver started at http://127.0.97.3:43579/ using document root <none> and password file <none>
I20260924 09:56:37.993106 8347 fs_manager.cc:359] Metadata directory not provided
I20260924 09:56:37.993207 8347 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 09:56:37.993491 8347 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 09:56:37.996212 8347 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790243735634835-388-0/minicluster-data/ts-2/data/instance:
uuid: "818b4870af8045b9ac5ed6fdd83d89dd"
format_stamp: "Formatted at 2026-09-24 09:56:37 on dist-test-slave-w8d2"
server_key: "6b0a5f898f9686379bedf49c4287a5c4"
server_key_iv: "9fde68ca08490d85d5c97ccb0d7bb445"
server_key_version: "encryptionkey@0"
I20260924 09:56:37.996903 8347 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790243735634835-388-0/minicluster-data/ts-2/wal/instance:
uuid: "818b4870af8045b9ac5ed6fdd83d89dd"
format_stamp: "Formatted at 2026-09-24 09:56:37 on dist-test-slave-w8d2"
server_key: "6b0a5f898f9686379bedf49c4287a5c4"
server_key_iv: "9fde68ca08490d85d5c97ccb0d7bb445"
server_key_version: "encryptionkey@0"
I20260924 09:56:38.001559 8347 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 09:56:38.004686 8362 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 09:56:38.005836 8347 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 09:56:38.005971 8347 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790243735634835-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790243735634835-388-0/minicluster-data/ts-2/wal
uuid: "818b4870af8045b9ac5ed6fdd83d89dd"
format_stamp: "Formatted at 2026-09-24 09:56:37 on dist-test-slave-w8d2"
server_key: "6b0a5f898f9686379bedf49c4287a5c4"
server_key_iv: "9fde68ca08490d85d5c97ccb0d7bb445"
server_key_version: "encryptionkey@0"
I20260924 09:56:38.006158 8347 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790243735634835-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790243735634835-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790243735634835-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 09:56:38.021389 8347 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 09:56:38.024919 8347 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 09:56:38.025151 8347 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 09:56:38.027607 8347 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 09:56:38.027686 8347 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 09:56:38.027829 8347 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 09:56:38.027885 8347 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 09:56:38 dist-test-slave-w8d2 krb5kdc[7953](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243797, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260924 09:56:38.049149 8046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:38.027995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:53409 (local address 127.0.97.62:38371)
0924 09:56:38.028214 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0924 09:56:38.028222 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 09:56:38.029498 (+ 1276us) server_negotiation.cc:407] Connection header received
0924 09:56:38.031599 (+ 2101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:56:38.031605 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:56:38.031694 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:56:38.031790 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:56:38.033980 (+ 2190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:38.034917 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:38.037216 (+ 2299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:38.037866 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:38.037996 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:38.042690 (+ 4694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:56:38.042697 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:56:38.042703 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:56:38.042749 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:56:38.045975 (+ 3226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:56:38.046639 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:56:38.046645 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:56:38.046649 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:56:38.046799 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:56:38.047339 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:56:38.047344 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:56:38.047347 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:56:38.047718 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:56:38.047877 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0924 09:56:38.048524 (+ 647us) server_negotiation.cc:299] Negotiation successful
0924 09:56:38.048769 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260924 09:56:38.049857 8371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:38.028556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38371 (local address 127.0.97.3:53409)
0924 09:56:38.029343 (+ 787us) negotiation.cc:107] Waiting for socket to connect
0924 09:56:38.029384 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 09:56:38.031325 (+ 1941us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:56:38.032168 (+ 843us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:56:38.032176 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:56:38.032931 (+ 755us) client_negotiation.cc:189] Negotiated authn=SASL
0924 09:56:38.033810 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:38.033835 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:38.035055 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:38.035061 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:38.037047 (+ 1986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:38.037062 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:38.038119 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:38.038126 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:38.038394 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:38.039394 (+ 1000us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 09:56:38.039435 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0924 09:56:38.042425 (+ 2990us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 09:56:38.046103 (+ 3678us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 09:56:38.046111 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 09:56:38.046128 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 09:56:38.046516 (+ 388us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 09:56:38.046943 (+ 427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 09:56:38.046949 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 09:56:38.046952 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 09:56:38.047198 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 09:56:38.047850 (+ 652us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 09:56:38.047858 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 09:56:38.048179 (+ 321us) client_negotiation.cc:769] Sending connection context
0924 09:56:38.048396 (+ 217us) client_negotiation.cc:240] Negotiation successful
0924 09:56:38.048770 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":523,"thread_start_us":315,"threads_started":1}
I20260924 09:56:38.071197 8046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:38.055733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:45673 (local address 127.0.97.62:38371)
0924 09:56:38.055887 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0924 09:56:38.055891 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 09:56:38.059777 (+ 3886us) server_negotiation.cc:407] Connection header received
0924 09:56:38.059925 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:56:38.059931 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:56:38.059995 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:56:38.060070 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:56:38.060912 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:38.061594 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:38.062811 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:38.063211 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:38.063341 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:38.065441 (+ 2100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:56:38.065447 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:56:38.065452 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:56:38.065480 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:56:38.068373 (+ 2893us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:56:38.069225 (+ 852us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:56:38.069228 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:56:38.069230 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:56:38.069379 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:56:38.069784 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:56:38.069789 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:56:38.069792 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:56:38.070117 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:56:38.070254 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0924 09:56:38.070657 (+ 403us) server_negotiation.cc:299] Negotiation successful
0924 09:56:38.070844 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 09:56:38.071446 8436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:38.055302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38371 (local address 127.0.97.3:45673)
0924 09:56:38.059674 (+ 4372us) negotiation.cc:107] Waiting for socket to connect
0924 09:56:38.059693 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 09:56:38.059821 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:56:38.060103 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:56:38.060107 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:56:38.060524 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0924 09:56:38.060785 (+ 261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:38.060794 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:38.061721 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:38.061726 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:38.062689 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:38.062699 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:38.063410 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:38.063416 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:38.063585 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:38.064221 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 09:56:38.064244 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 09:56:38.065290 (+ 1046us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 09:56:38.068544 (+ 3254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 09:56:38.068549 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 09:56:38.068552 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 09:56:38.069077 (+ 525us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 09:56:38.069526 (+ 449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 09:56:38.069531 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 09:56:38.069538 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0924 09:56:38.069687 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 09:56:38.070225 (+ 538us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 09:56:38.070232 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 09:56:38.070412 (+ 180us) client_negotiation.cc:769] Sending connection context
0924 09:56:38.070549 (+ 137us) client_negotiation.cc:240] Negotiation successful
0924 09:56:38.070908 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4228,"thread_start_us":4086,"threads_started":1}
I20260924 09:56:38.079276 8347 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:36899
I20260924 09:56:38.079340 8481 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:36899 every 8 connection(s)
I20260924 09:56:38.080408 8347 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790243735634835-388-0/minicluster-data/ts-2/data/info.pb
I20260924 09:56:38.082089 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu as pid 8347
I20260924 09:56:38.082269 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790243735634835-388-0/minicluster-data/ts-2/wal/instance
I20260924 09:56:38.082710 388 external_mini_cluster.cc:1428] Setting key 412075a3a5bcac1db1c7deb668ad8fee
I20260924 09:56:38.082206 8482 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38371
I20260924 09:56:38.083396 8482 heartbeater.cc:464] Registering TS with master...
I20260924 09:56:38.084020 8482 heartbeater.cc:511] Master 127.0.97.62:38371 requested a full tablet report, sending...
I20260924 09:56:38.085147 7998 ts_manager.cc:194] Registered new tserver with Master: 818b4870af8045b9ac5ed6fdd83d89dd (127.0.97.3:36899)
I20260924 09:56:38.085902 7998 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:53409
I20260924 09:56:38.096843 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 09:56:38 dist-test-slave-w8d2 krb5kdc[7953](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243798, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 09:56:38 dist-test-slave-w8d2 krb5kdc[7953](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 09:56:38 dist-test-slave-w8d2 krb5kdc[7953](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260924 09:56:38.143934 8046 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 09:56:38.133774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36686 (local address 127.0.97.62:38371)
0924 09:56:38.133983 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0924 09:56:38.133989 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 09:56:38.134308 (+ 319us) server_negotiation.cc:407] Connection header received
0924 09:56:38.134450 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:56:38.134456 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:56:38.134536 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:56:38.134648 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:56:38.135922 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:38.137041 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:38.138350 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:38.138815 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:38.138924 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:38.143676 (+ 4752us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36686: BlockingRecv error: recv got EOF from 127.0.0.1:36686 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
I20260924 09:56:38.144888 8204 heartbeater.cc:503] Master 127.0.97.62:38371 was elected leader, sending a full tablet report...
Sep 24 09:56:38 dist-test-slave-w8d2 krb5kdc[7953](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243798, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260924 09:56:38.159109 8046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:38.147002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36702 (local address 127.0.97.62:38371)
0924 09:56:38.147139 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0924 09:56:38.147143 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 09:56:38.147463 (+ 320us) server_negotiation.cc:407] Connection header received
0924 09:56:38.147581 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:56:38.147585 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:56:38.147650 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:56:38.147725 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:56:38.148687 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:38.149358 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:38.150453 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:38.150858 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:38.150961 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:38.154523 (+ 3562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:56:38.154529 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:56:38.154534 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:56:38.154566 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:56:38.156797 (+ 2231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:56:38.157394 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:56:38.157398 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:56:38.157400 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:56:38.157507 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:56:38.157831 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:56:38.157836 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:56:38.157839 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:56:38.158097 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:56:38.158310 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0924 09:56:38.158592 (+ 282us) server_negotiation.cc:299] Negotiation successful
0924 09:56:38.158790 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 09:56:38.162631 7998 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36702:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 09:56:38.169253 7998 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 09:56:38.201972 8502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:38.180515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40363 (local address 127.0.0.1:38830)
0924 09:56:38.190987 (+ 10472us) negotiation.cc:107] Waiting for socket to connect
0924 09:56:38.191059 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0924 09:56:38.192695 (+ 1636us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:56:38.193741 (+ 1046us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:56:38.193749 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:56:38.193780 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 09:56:38.194262 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:38.194274 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:38.196325 (+ 2051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:38.196331 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:38.198964 (+ 2633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:38.198978 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:38.201070 (+ 2092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:38.201076 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:38.201276 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:38.201327 (+ 51us) client_negotiation.cc:769] Sending connection context
0924 09:56:38.201451 (+ 124us) client_negotiation.cc:240] Negotiation successful
0924 09:56:38.201626 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10153,"thread_start_us":16638,"threads_started":1}
I20260924 09:56:38.202348 8501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:38.180701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38830 (local address 127.0.97.1:40363)
0924 09:56:38.181238 (+ 537us) server_negotiation.cc:206] Beginning negotiation
0924 09:56:38.181248 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 09:56:38.191207 (+ 9959us) server_negotiation.cc:407] Connection header received
0924 09:56:38.192973 (+ 1766us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:56:38.192983 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:56:38.193288 (+ 305us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:56:38.193451 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 09:56:38.194464 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:38.196115 (+ 1651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:38.199140 (+ 3025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:38.200622 (+ 1482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:38.200747 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:38.200946 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0924 09:56:38.201525 (+ 579us) server_negotiation.cc:299] Negotiation successful
0924 09:56:38.201686 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":381,"thread_start_us":821,"threads_started":1}
I20260924 09:56:38.204034 8500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:38.180983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:36899 (local address 127.0.0.1:50590)
0924 09:56:38.196907 (+ 15924us) negotiation.cc:107] Waiting for socket to connect
0924 09:56:38.196930 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 09:56:38.197066 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:56:38.197633 (+ 567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:56:38.197637 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:56:38.197665 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 09:56:38.198052 (+ 387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:38.198062 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:38.199513 (+ 1451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:38.199519 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:38.201577 (+ 2058us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:38.201589 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:38.203288 (+ 1699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:38.203295 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:38.203454 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:38.203486 (+ 32us) client_negotiation.cc:769] Sending connection context
0924 09:56:38.203590 (+ 104us) client_negotiation.cc:240] Negotiation successful
0924 09:56:38.203706 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15772,"thread_start_us":9519,"threads_started":1}
I20260924 09:56:38.204689 8504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:38.181334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50590 (local address 127.0.97.3:36899)
0924 09:56:38.192353 (+ 11019us) server_negotiation.cc:206] Beginning negotiation
0924 09:56:38.192363 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 09:56:38.197046 (+ 4683us) server_negotiation.cc:407] Connection header received
0924 09:56:38.197216 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:56:38.197224 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:56:38.197450 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:56:38.197642 (+ 192us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 09:56:38.198223 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:38.199358 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:38.201740 (+ 2382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:38.203069 (+ 1329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:38.203186 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:38.203415 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0924 09:56:38.204223 (+ 808us) server_negotiation.cc:299] Negotiation successful
0924 09:56:38.204371 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10833,"thread_start_us":11030,"threads_started":1}
I20260924 09:56:38.206233 8503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:38.181085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:41195 (local address 127.0.0.1:36820)
0924 09:56:38.199486 (+ 18401us) negotiation.cc:107] Waiting for socket to connect
0924 09:56:38.199509 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 09:56:38.199613 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:56:38.200204 (+ 591us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:56:38.200209 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:56:38.200231 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 09:56:38.200599 (+ 368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:38.200608 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:38.201835 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:38.201840 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:38.203928 (+ 2088us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:38.203939 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:38.205550 (+ 1611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:38.205556 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:38.205714 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:38.205743 (+ 29us) client_negotiation.cc:769] Sending connection context
0924 09:56:38.205834 (+ 91us) client_negotiation.cc:240] Negotiation successful
0924 09:56:38.205948 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18296,"thread_start_us":18165,"threads_started":1}
I20260924 09:56:38.206303 8505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:38.181535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36820 (local address 127.0.97.2:41195)
0924 09:56:38.196726 (+ 15191us) server_negotiation.cc:206] Beginning negotiation
0924 09:56:38.196734 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 09:56:38.199739 (+ 3005us) server_negotiation.cc:407] Connection header received
0924 09:56:38.199845 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:56:38.199852 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:56:38.200037 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:56:38.200208 (+ 171us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 09:56:38.200747 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:38.201679 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:38.204105 (+ 2426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:38.205372 (+ 1267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:38.205477 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:38.205664 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0924 09:56:38.205891 (+ 227us) server_negotiation.cc:299] Negotiation successful
0924 09:56:38.206042 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15005,"thread_start_us":15180,"threads_started":1}
I20260924 09:56:38.206156 8134 tablet_service.cc:1467] Processing CreateTablet for tablet 52fb6058be404bb5ab25604fc2aba32c (DEFAULT_TABLE table=test-table [id=99527f83296b4d7fbbbbf27eab316731]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 09:56:38.208142 8134 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 52fb6058be404bb5ab25604fc2aba32c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 09:56:38.208299 8412 tablet_service.cc:1467] Processing CreateTablet for tablet 52fb6058be404bb5ab25604fc2aba32c (DEFAULT_TABLE table=test-table [id=99527f83296b4d7fbbbbf27eab316731]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 09:56:38.209613 8412 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 52fb6058be404bb5ab25604fc2aba32c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 09:56:38.214339 8277 tablet_service.cc:1467] Processing CreateTablet for tablet 52fb6058be404bb5ab25604fc2aba32c (DEFAULT_TABLE table=test-table [id=99527f83296b4d7fbbbbf27eab316731]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 09:56:38.215648 8277 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 52fb6058be404bb5ab25604fc2aba32c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 09:56:38.219760 8506 tablet_bootstrap.cc:492] T 52fb6058be404bb5ab25604fc2aba32c P 818b4870af8045b9ac5ed6fdd83d89dd: Bootstrap starting.
I20260924 09:56:38.221477 8507 tablet_bootstrap.cc:492] T 52fb6058be404bb5ab25604fc2aba32c P 3e6632b5e12e420283aca499e3c94c42: Bootstrap starting.
I20260924 09:56:38.223104 8506 tablet_bootstrap.cc:654] T 52fb6058be404bb5ab25604fc2aba32c P 818b4870af8045b9ac5ed6fdd83d89dd: Neither blocks nor log segments found. Creating new log.
I20260924 09:56:38.224279 8506 log.cc:824] T 52fb6058be404bb5ab25604fc2aba32c P 818b4870af8045b9ac5ed6fdd83d89dd: Log is configured to *not* fsync() on all Append() calls
I20260924 09:56:38.224574 8507 tablet_bootstrap.cc:654] T 52fb6058be404bb5ab25604fc2aba32c P 3e6632b5e12e420283aca499e3c94c42: Neither blocks nor log segments found. Creating new log.
I20260924 09:56:38.225579 8507 log.cc:824] T 52fb6058be404bb5ab25604fc2aba32c P 3e6632b5e12e420283aca499e3c94c42: Log is configured to *not* fsync() on all Append() calls
I20260924 09:56:38.227007 8506 tablet_bootstrap.cc:492] T 52fb6058be404bb5ab25604fc2aba32c P 818b4870af8045b9ac5ed6fdd83d89dd: No bootstrap required, opened a new log
I20260924 09:56:38.227274 8506 ts_tablet_manager.cc:1397] T 52fb6058be404bb5ab25604fc2aba32c P 818b4870af8045b9ac5ed6fdd83d89dd: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260924 09:56:38.228263 8507 tablet_bootstrap.cc:492] T 52fb6058be404bb5ab25604fc2aba32c P 3e6632b5e12e420283aca499e3c94c42: No bootstrap required, opened a new log
I20260924 09:56:38.228554 8507 ts_tablet_manager.cc:1397] T 52fb6058be404bb5ab25604fc2aba32c P 3e6632b5e12e420283aca499e3c94c42: Time spent bootstrapping tablet: real 0.007s user 0.007s sys 0.000s
I20260924 09:56:38.229589 8510 tablet_bootstrap.cc:492] T 52fb6058be404bb5ab25604fc2aba32c P 43f12639d53d4d19b96ec67379f6fa1f: Bootstrap starting.
I20260924 09:56:38.232040 8510 tablet_bootstrap.cc:654] T 52fb6058be404bb5ab25604fc2aba32c P 43f12639d53d4d19b96ec67379f6fa1f: Neither blocks nor log segments found. Creating new log.
I20260924 09:56:38.232991 8510 log.cc:824] T 52fb6058be404bb5ab25604fc2aba32c P 43f12639d53d4d19b96ec67379f6fa1f: Log is configured to *not* fsync() on all Append() calls
I20260924 09:56:38.235126 8510 tablet_bootstrap.cc:492] T 52fb6058be404bb5ab25604fc2aba32c P 43f12639d53d4d19b96ec67379f6fa1f: No bootstrap required, opened a new log
I20260924 09:56:38.235373 8510 ts_tablet_manager.cc:1397] T 52fb6058be404bb5ab25604fc2aba32c P 43f12639d53d4d19b96ec67379f6fa1f: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260924 09:56:38.237138 8506 raft_consensus.cc:348] T 52fb6058be404bb5ab25604fc2aba32c P 818b4870af8045b9ac5ed6fdd83d89dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e6632b5e12e420283aca499e3c94c42" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40363 } } peers { permanent_uuid: "818b4870af8045b9ac5ed6fdd83d89dd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36899 } } peers { permanent_uuid: "43f12639d53d4d19b96ec67379f6fa1f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41195 } }
I20260924 09:56:38.237634 8506 raft_consensus.cc:374] T 52fb6058be404bb5ab25604fc2aba32c P 818b4870af8045b9ac5ed6fdd83d89dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 09:56:38.237772 8506 raft_consensus.cc:729] T 52fb6058be404bb5ab25604fc2aba32c P 818b4870af8045b9ac5ed6fdd83d89dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 818b4870af8045b9ac5ed6fdd83d89dd, State: Initialized, Role: FOLLOWER
I20260924 09:56:38.238276 8507 raft_consensus.cc:348] T 52fb6058be404bb5ab25604fc2aba32c P 3e6632b5e12e420283aca499e3c94c42 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e6632b5e12e420283aca499e3c94c42" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40363 } } peers { permanent_uuid: "818b4870af8045b9ac5ed6fdd83d89dd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36899 } } peers { permanent_uuid: "43f12639d53d4d19b96ec67379f6fa1f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41195 } }
I20260924 09:56:38.238646 8507 raft_consensus.cc:374] T 52fb6058be404bb5ab25604fc2aba32c P 3e6632b5e12e420283aca499e3c94c42 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 09:56:38.238502 8506 consensus_queue.cc:260] T 52fb6058be404bb5ab25604fc2aba32c P 818b4870af8045b9ac5ed6fdd83d89dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e6632b5e12e420283aca499e3c94c42" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40363 } } peers { permanent_uuid: "818b4870af8045b9ac5ed6fdd83d89dd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36899 } } peers { permanent_uuid: "43f12639d53d4d19b96ec67379f6fa1f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41195 } }
I20260924 09:56:38.238732 8507 raft_consensus.cc:729] T 52fb6058be404bb5ab25604fc2aba32c P 3e6632b5e12e420283aca499e3c94c42 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e6632b5e12e420283aca499e3c94c42, State: Initialized, Role: FOLLOWER
I20260924 09:56:38.239689 8482 heartbeater.cc:503] Master 127.0.97.62:38371 was elected leader, sending a full tablet report...
I20260924 09:56:38.240228 8506 ts_tablet_manager.cc:1428] T 52fb6058be404bb5ab25604fc2aba32c P 818b4870af8045b9ac5ed6fdd83d89dd: Time spent starting tablet: real 0.013s user 0.009s sys 0.003s
I20260924 09:56:38.239452 8507 consensus_queue.cc:260] T 52fb6058be404bb5ab25604fc2aba32c P 3e6632b5e12e420283aca499e3c94c42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e6632b5e12e420283aca499e3c94c42" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40363 } } peers { permanent_uuid: "818b4870af8045b9ac5ed6fdd83d89dd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36899 } } peers { permanent_uuid: "43f12639d53d4d19b96ec67379f6fa1f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41195 } }
I20260924 09:56:38.241758 8507 ts_tablet_manager.cc:1428] T 52fb6058be404bb5ab25604fc2aba32c P 3e6632b5e12e420283aca499e3c94c42: Time spent starting tablet: real 0.013s user 0.011s sys 0.000s
I20260924 09:56:38.244191 8510 raft_consensus.cc:348] T 52fb6058be404bb5ab25604fc2aba32c P 43f12639d53d4d19b96ec67379f6fa1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e6632b5e12e420283aca499e3c94c42" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40363 } } peers { permanent_uuid: "818b4870af8045b9ac5ed6fdd83d89dd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36899 } } peers { permanent_uuid: "43f12639d53d4d19b96ec67379f6fa1f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41195 } }
I20260924 09:56:38.244536 8510 raft_consensus.cc:374] T 52fb6058be404bb5ab25604fc2aba32c P 43f12639d53d4d19b96ec67379f6fa1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 09:56:38.244625 8510 raft_consensus.cc:729] T 52fb6058be404bb5ab25604fc2aba32c P 43f12639d53d4d19b96ec67379f6fa1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43f12639d53d4d19b96ec67379f6fa1f, State: Initialized, Role: FOLLOWER
I20260924 09:56:38.245127 8510 consensus_queue.cc:260] T 52fb6058be404bb5ab25604fc2aba32c P 43f12639d53d4d19b96ec67379f6fa1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e6632b5e12e420283aca499e3c94c42" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40363 } } peers { permanent_uuid: "818b4870af8045b9ac5ed6fdd83d89dd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36899 } } peers { permanent_uuid: "43f12639d53d4d19b96ec67379f6fa1f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41195 } }
I20260924 09:56:38.246124 8343 heartbeater.cc:503] Master 127.0.97.62:38371 was elected leader, sending a full tablet report...
I20260924 09:56:38.246551 8510 ts_tablet_manager.cc:1428] T 52fb6058be404bb5ab25604fc2aba32c P 43f12639d53d4d19b96ec67379f6fa1f: Time spent starting tablet: real 0.011s user 0.009s sys 0.000s
W20260924 09:56:38.331899 8483 tablet.cc:2369] T 52fb6058be404bb5ab25604fc2aba32c P 818b4870af8045b9ac5ed6fdd83d89dd: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 09:56:38.357630 8344 tablet.cc:2369] T 52fb6058be404bb5ab25604fc2aba32c P 43f12639d53d4d19b96ec67379f6fa1f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 09:56:38.387380 8205 tablet.cc:2369] T 52fb6058be404bb5ab25604fc2aba32c P 3e6632b5e12e420283aca499e3c94c42: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 09:56:38.420166 8512 raft_consensus.cc:482] T 52fb6058be404bb5ab25604fc2aba32c P 818b4870af8045b9ac5ed6fdd83d89dd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 09:56:38.420485 8512 raft_consensus.cc:504] T 52fb6058be404bb5ab25604fc2aba32c P 818b4870af8045b9ac5ed6fdd83d89dd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e6632b5e12e420283aca499e3c94c42" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40363 } } peers { permanent_uuid: "818b4870af8045b9ac5ed6fdd83d89dd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36899 } } peers { permanent_uuid: "43f12639d53d4d19b96ec67379f6fa1f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41195 } }
I20260924 09:56:38.422072 8512 leader_election.cc:288] T 52fb6058be404bb5ab25604fc2aba32c P 818b4870af8045b9ac5ed6fdd83d89dd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3e6632b5e12e420283aca499e3c94c42 (127.0.97.1:40363), 43f12639d53d4d19b96ec67379f6fa1f (127.0.97.2:41195)
I20260924 09:56:38.427287 8505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:38.422467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:47343 (local address 127.0.97.2:41195)
0924 09:56:38.422748 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0924 09:56:38.422755 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 09:56:38.422776 (+ 21us) server_negotiation.cc:407] Connection header received
0924 09:56:38.422929 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:56:38.422933 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:56:38.422988 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:56:38.423058 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 09:56:38.423452 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:38.424157 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:38.425723 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:38.426480 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:38.426600 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:38.426698 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0924 09:56:38.426923 (+ 225us) server_negotiation.cc:299] Negotiation successful
0924 09:56:38.427034 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138}
I20260924 09:56:38.427317 8371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:38.422395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:41195 (local address 127.0.97.3:47343)
0924 09:56:38.422679 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0924 09:56:38.422695 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 09:56:38.422789 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:56:38.423103 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:56:38.423108 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:56:38.423132 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 09:56:38.423334 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:38.423343 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:38.424273 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:38.424278 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:38.425588 (+ 1310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:38.425602 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:38.426625 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:38.426630 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:38.426790 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:38.426814 (+ 24us) client_negotiation.cc:769] Sending connection context
0924 09:56:38.426890 (+ 76us) client_negotiation.cc:240] Negotiation successful
0924 09:56:38.427010 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"mutex_wait_us":1}
I20260924 09:56:38.428081 8501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:38.422381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:37375 (local address 127.0.97.1:40363)
0924 09:56:38.422574 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0924 09:56:38.422580 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 09:56:38.423341 (+ 761us) server_negotiation.cc:407] Connection header received
0924 09:56:38.423501 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:56:38.423505 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:56:38.423557 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:56:38.423627 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 09:56:38.424085 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:38.424830 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:38.426448 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:38.427188 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:38.427341 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:38.427430 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0924 09:56:38.427745 (+ 315us) server_negotiation.cc:299] Negotiation successful
0924 09:56:38.427866 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 09:56:38.428151 8515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:38.422416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40363 (local address 127.0.97.3:37375)
0924 09:56:38.423217 (+ 801us) negotiation.cc:107] Waiting for socket to connect
0924 09:56:38.423255 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 09:56:38.423386 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:56:38.423651 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:56:38.423655 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:56:38.423686 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 09:56:38.423960 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:38.423968 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:38.424945 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:38.424950 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:38.426312 (+ 1362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:38.426322 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:38.427419 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:38.427424 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:38.427594 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:38.427626 (+ 32us) client_negotiation.cc:769] Sending connection context
0924 09:56:38.427707 (+ 81us) client_negotiation.cc:240] Negotiation successful
0924 09:56:38.427818 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":701,"mutex_wait_us":4,"thread_start_us":516,"threads_started":1}
I20260924 09:56:38.428224 8298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52fb6058be404bb5ab25604fc2aba32c" candidate_uuid: "818b4870af8045b9ac5ed6fdd83d89dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43f12639d53d4d19b96ec67379f6fa1f" is_pre_election: true
I20260924 09:56:38.428731 8298 raft_consensus.cc:2454] T 52fb6058be404bb5ab25604fc2aba32c P 43f12639d53d4d19b96ec67379f6fa1f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818b4870af8045b9ac5ed6fdd83d89dd in term 0.
I20260924 09:56:38.429021 8154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52fb6058be404bb5ab25604fc2aba32c" candidate_uuid: "818b4870af8045b9ac5ed6fdd83d89dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e6632b5e12e420283aca499e3c94c42" is_pre_election: true
I20260924 09:56:38.429346 8363 leader_election.cc:302] T 52fb6058be404bb5ab25604fc2aba32c P 818b4870af8045b9ac5ed6fdd83d89dd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43f12639d53d4d19b96ec67379f6fa1f, 818b4870af8045b9ac5ed6fdd83d89dd; no voters:
I20260924 09:56:38.429446 8154 raft_consensus.cc:2454] T 52fb6058be404bb5ab25604fc2aba32c P 3e6632b5e12e420283aca499e3c94c42 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818b4870af8045b9ac5ed6fdd83d89dd in term 0.
I20260924 09:56:38.429677 8512 raft_consensus.cc:2781] T 52fb6058be404bb5ab25604fc2aba32c P 818b4870af8045b9ac5ed6fdd83d89dd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 09:56:38.429814 8512 raft_consensus.cc:482] T 52fb6058be404bb5ab25604fc2aba32c P 818b4870af8045b9ac5ed6fdd83d89dd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 09:56:38.429917 8512 raft_consensus.cc:3037] T 52fb6058be404bb5ab25604fc2aba32c P 818b4870af8045b9ac5ed6fdd83d89dd [term 0 FOLLOWER]: Advancing to term 1
I20260924 09:56:38.432039 8512 raft_consensus.cc:504] T 52fb6058be404bb5ab25604fc2aba32c P 818b4870af8045b9ac5ed6fdd83d89dd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e6632b5e12e420283aca499e3c94c42" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40363 } } peers { permanent_uuid: "818b4870af8045b9ac5ed6fdd83d89dd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36899 } } peers { permanent_uuid: "43f12639d53d4d19b96ec67379f6fa1f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41195 } }
I20260924 09:56:38.432648 8512 leader_election.cc:288] T 52fb6058be404bb5ab25604fc2aba32c P 818b4870af8045b9ac5ed6fdd83d89dd [CANDIDATE]: Term 1 election: Requested vote from peers 3e6632b5e12e420283aca499e3c94c42 (127.0.97.1:40363), 43f12639d53d4d19b96ec67379f6fa1f (127.0.97.2:41195)
I20260924 09:56:38.432974 8154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52fb6058be404bb5ab25604fc2aba32c" candidate_uuid: "818b4870af8045b9ac5ed6fdd83d89dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e6632b5e12e420283aca499e3c94c42"
I20260924 09:56:38.433058 8298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52fb6058be404bb5ab25604fc2aba32c" candidate_uuid: "818b4870af8045b9ac5ed6fdd83d89dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43f12639d53d4d19b96ec67379f6fa1f"
I20260924 09:56:38.433167 8154 raft_consensus.cc:3037] T 52fb6058be404bb5ab25604fc2aba32c P 3e6632b5e12e420283aca499e3c94c42 [term 0 FOLLOWER]: Advancing to term 1
I20260924 09:56:38.433243 8298 raft_consensus.cc:3037] T 52fb6058be404bb5ab25604fc2aba32c P 43f12639d53d4d19b96ec67379f6fa1f [term 0 FOLLOWER]: Advancing to term 1
I20260924 09:56:38.435305 8154 raft_consensus.cc:2454] T 52fb6058be404bb5ab25604fc2aba32c P 3e6632b5e12e420283aca499e3c94c42 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 818b4870af8045b9ac5ed6fdd83d89dd in term 1.
I20260924 09:56:38.435321 8298 raft_consensus.cc:2454] T 52fb6058be404bb5ab25604fc2aba32c P 43f12639d53d4d19b96ec67379f6fa1f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 818b4870af8045b9ac5ed6fdd83d89dd in term 1.
I20260924 09:56:38.435607 8363 leader_election.cc:302] T 52fb6058be404bb5ab25604fc2aba32c P 818b4870af8045b9ac5ed6fdd83d89dd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43f12639d53d4d19b96ec67379f6fa1f, 818b4870af8045b9ac5ed6fdd83d89dd; no voters:
I20260924 09:56:38.435858 8512 raft_consensus.cc:2781] T 52fb6058be404bb5ab25604fc2aba32c P 818b4870af8045b9ac5ed6fdd83d89dd [term 1 FOLLOWER]: Leader election won for term 1
I20260924 09:56:38.436218 8512 raft_consensus.cc:686] T 52fb6058be404bb5ab25604fc2aba32c P 818b4870af8045b9ac5ed6fdd83d89dd [term 1 LEADER]: Becoming Leader. State: Replica: 818b4870af8045b9ac5ed6fdd83d89dd, State: Running, Role: LEADER
I20260924 09:56:38.436646 8512 consensus_queue.cc:237] T 52fb6058be404bb5ab25604fc2aba32c P 818b4870af8045b9ac5ed6fdd83d89dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e6632b5e12e420283aca499e3c94c42" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40363 } } peers { permanent_uuid: "818b4870af8045b9ac5ed6fdd83d89dd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36899 } } peers { permanent_uuid: "43f12639d53d4d19b96ec67379f6fa1f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41195 } }
I20260924 09:56:38.440924 7997 catalog_manager.cc:5495] T 52fb6058be404bb5ab25604fc2aba32c P 818b4870af8045b9ac5ed6fdd83d89dd reported cstate change: term changed from 0 to 1, leader changed from <none> to 818b4870af8045b9ac5ed6fdd83d89dd (127.0.97.3). New cstate: current_term: 1 leader_uuid: "818b4870af8045b9ac5ed6fdd83d89dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e6632b5e12e420283aca499e3c94c42" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40363 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "818b4870af8045b9ac5ed6fdd83d89dd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36899 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "43f12639d53d4d19b96ec67379f6fa1f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41195 } health_report { overall_health: UNKNOWN } } }
I20260924 09:56:38.501645 8504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:38.496577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50594 (local address 127.0.97.3:36899)
0924 09:56:38.496772 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0924 09:56:38.496778 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 09:56:38.496795 (+ 17us) server_negotiation.cc:407] Connection header received
0924 09:56:38.496962 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:56:38.496965 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:56:38.497049 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:56:38.497194 (+ 145us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 09:56:38.497522 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:38.498215 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:38.499500 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:38.500008 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:38.500132 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:38.500605 (+ 473us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 09:56:38.501169 (+ 564us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 09:56:38.501270 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0924 09:56:38.501337 (+ 67us) server_negotiation.cc:299] Negotiation successful
0924 09:56:38.501395 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260924 09:56:38.509068 8154 raft_consensus.cc:1261] T 52fb6058be404bb5ab25604fc2aba32c P 3e6632b5e12e420283aca499e3c94c42 [term 1 FOLLOWER]: Refusing update from remote peer 818b4870af8045b9ac5ed6fdd83d89dd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 09:56:38.509068 8298 raft_consensus.cc:1261] T 52fb6058be404bb5ab25604fc2aba32c P 43f12639d53d4d19b96ec67379f6fa1f [term 1 FOLLOWER]: Refusing update from remote peer 818b4870af8045b9ac5ed6fdd83d89dd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 09:56:38.509928 8516 consensus_queue.cc:1035] T 52fb6058be404bb5ab25604fc2aba32c P 818b4870af8045b9ac5ed6fdd83d89dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "43f12639d53d4d19b96ec67379f6fa1f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 09:56:38.510277 8512 consensus_queue.cc:1035] T 52fb6058be404bb5ab25604fc2aba32c P 818b4870af8045b9ac5ed6fdd83d89dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e6632b5e12e420283aca499e3c94c42" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40363 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 09:56:38.533059 7998 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36702:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 09:56:38.533334 7998 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36702:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 09:56:38.536720 7998 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e3f28e3acf1e44e59f3ed21ee9c9bd10: Sending DeleteTablet for 3 replicas of tablet 52fb6058be404bb5ab25604fc2aba32c
I20260924 09:56:38.537737 8134 tablet_service.cc:1514] Processing DeleteTablet for tablet 52fb6058be404bb5ab25604fc2aba32c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 09:56:38 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:38830
I20260924 09:56:38.538058 8277 tablet_service.cc:1514] Processing DeleteTablet for tablet 52fb6058be404bb5ab25604fc2aba32c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 09:56:38 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:36820
I20260924 09:56:38.538416 8412 tablet_service.cc:1514] Processing DeleteTablet for tablet 52fb6058be404bb5ab25604fc2aba32c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 09:56:38 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:50590
I20260924 09:56:38.538674 8528 tablet_replica.cc:330] T 52fb6058be404bb5ab25604fc2aba32c P 3e6632b5e12e420283aca499e3c94c42: stopping tablet replica
I20260924 09:56:38.539142 8528 raft_consensus.cc:2229] T 52fb6058be404bb5ab25604fc2aba32c P 3e6632b5e12e420283aca499e3c94c42 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 09:56:38.539697 8528 raft_consensus.cc:2258] T 52fb6058be404bb5ab25604fc2aba32c P 3e6632b5e12e420283aca499e3c94c42 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 09:56:38.540719 7998 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36702:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260924 09:56:38.541745 8528 ts_tablet_manager.cc:1905] T 52fb6058be404bb5ab25604fc2aba32c P 3e6632b5e12e420283aca499e3c94c42: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260924 09:56:38.542258 7998 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 09:56:38.547865 8530 tablet_replica.cc:330] T 52fb6058be404bb5ab25604fc2aba32c P 818b4870af8045b9ac5ed6fdd83d89dd: stopping tablet replica
I20260924 09:56:38.548403 8530 raft_consensus.cc:2229] T 52fb6058be404bb5ab25604fc2aba32c P 818b4870af8045b9ac5ed6fdd83d89dd [term 1 LEADER]: Raft consensus shutting down.
I20260924 09:56:38.548815 8528 ts_tablet_manager.cc:1918] T 52fb6058be404bb5ab25604fc2aba32c P 3e6632b5e12e420283aca499e3c94c42: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 09:56:38.548991 8528 log.cc:1196] T 52fb6058be404bb5ab25604fc2aba32c P 3e6632b5e12e420283aca499e3c94c42: Deleting WAL directory at /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790243735634835-388-0/minicluster-data/ts-0/wal/wals/52fb6058be404bb5ab25604fc2aba32c
I20260924 09:56:38.549232 8530 raft_consensus.cc:2258] T 52fb6058be404bb5ab25604fc2aba32c P 818b4870af8045b9ac5ed6fdd83d89dd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 09:56:38.549511 8528 ts_tablet_manager.cc:1939] T 52fb6058be404bb5ab25604fc2aba32c P 3e6632b5e12e420283aca499e3c94c42: Deleting consensus metadata
I20260924 09:56:38.550034 8529 tablet_replica.cc:330] T 52fb6058be404bb5ab25604fc2aba32c P 43f12639d53d4d19b96ec67379f6fa1f: stopping tablet replica
I20260924 09:56:38.551177 7983 catalog_manager.cc:4841] TS 3e6632b5e12e420283aca499e3c94c42 (127.0.97.1:40363): tablet 52fb6058be404bb5ab25604fc2aba32c (table test-table [id=99527f83296b4d7fbbbbf27eab316731]) successfully deleted
I20260924 09:56:38.551414 8134 tablet_service.cc:1467] Processing CreateTablet for tablet a74b4d78d73a47d4a3d849053b2d454d (DEFAULT_TABLE table=test-table [id=8c55507b7cde4cffbfa98ab00e5bec29]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 09:56:38.551632 8529 raft_consensus.cc:2229] T 52fb6058be404bb5ab25604fc2aba32c P 43f12639d53d4d19b96ec67379f6fa1f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 09:56:38.551828 8134 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a74b4d78d73a47d4a3d849053b2d454d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 09:56:38.552119 8529 raft_consensus.cc:2258] T 52fb6058be404bb5ab25604fc2aba32c P 43f12639d53d4d19b96ec67379f6fa1f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 09:56:38.552948 8530 ts_tablet_manager.cc:1905] T 52fb6058be404bb5ab25604fc2aba32c P 818b4870af8045b9ac5ed6fdd83d89dd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 09:56:38.554042 8529 ts_tablet_manager.cc:1905] T 52fb6058be404bb5ab25604fc2aba32c P 43f12639d53d4d19b96ec67379f6fa1f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 09:56:38.554769 8272 tablet_service.cc:1467] Processing CreateTablet for tablet a74b4d78d73a47d4a3d849053b2d454d (DEFAULT_TABLE table=test-table [id=8c55507b7cde4cffbfa98ab00e5bec29]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 09:56:38.555141 8272 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a74b4d78d73a47d4a3d849053b2d454d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 09:56:38.558229 8507 tablet_bootstrap.cc:492] T a74b4d78d73a47d4a3d849053b2d454d P 3e6632b5e12e420283aca499e3c94c42: Bootstrap starting.
I20260924 09:56:38.558506 8412 tablet_service.cc:1467] Processing CreateTablet for tablet a74b4d78d73a47d4a3d849053b2d454d (DEFAULT_TABLE table=test-table [id=8c55507b7cde4cffbfa98ab00e5bec29]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 09:56:38.558879 8412 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a74b4d78d73a47d4a3d849053b2d454d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 09:56:38.559718 8507 tablet_bootstrap.cc:654] T a74b4d78d73a47d4a3d849053b2d454d P 3e6632b5e12e420283aca499e3c94c42: Neither blocks nor log segments found. Creating new log.
I20260924 09:56:38.559729 8530 ts_tablet_manager.cc:1918] T 52fb6058be404bb5ab25604fc2aba32c P 818b4870af8045b9ac5ed6fdd83d89dd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 09:56:38.559875 8530 log.cc:1196] T 52fb6058be404bb5ab25604fc2aba32c P 818b4870af8045b9ac5ed6fdd83d89dd: Deleting WAL directory at /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790243735634835-388-0/minicluster-data/ts-2/wal/wals/52fb6058be404bb5ab25604fc2aba32c
I20260924 09:56:38.560410 8530 ts_tablet_manager.cc:1939] T 52fb6058be404bb5ab25604fc2aba32c P 818b4870af8045b9ac5ed6fdd83d89dd: Deleting consensus metadata
I20260924 09:56:38.561142 8510 tablet_bootstrap.cc:492] T a74b4d78d73a47d4a3d849053b2d454d P 43f12639d53d4d19b96ec67379f6fa1f: Bootstrap starting.
I20260924 09:56:38.562681 8510 tablet_bootstrap.cc:654] T a74b4d78d73a47d4a3d849053b2d454d P 43f12639d53d4d19b96ec67379f6fa1f: Neither blocks nor log segments found. Creating new log.
I20260924 09:56:38.564828 8506 tablet_bootstrap.cc:492] T a74b4d78d73a47d4a3d849053b2d454d P 818b4870af8045b9ac5ed6fdd83d89dd: Bootstrap starting.
I20260924 09:56:38.566454 8506 tablet_bootstrap.cc:654] T a74b4d78d73a47d4a3d849053b2d454d P 818b4870af8045b9ac5ed6fdd83d89dd: Neither blocks nor log segments found. Creating new log.
I20260924 09:56:38.567902 7985 catalog_manager.cc:4841] TS 818b4870af8045b9ac5ed6fdd83d89dd (127.0.97.3:36899): tablet 52fb6058be404bb5ab25604fc2aba32c (table test-table [id=99527f83296b4d7fbbbbf27eab316731]) successfully deleted
I20260924 09:56:38.570652 8510 tablet_bootstrap.cc:492] T a74b4d78d73a47d4a3d849053b2d454d P 43f12639d53d4d19b96ec67379f6fa1f: No bootstrap required, opened a new log
I20260924 09:56:38.570842 8510 ts_tablet_manager.cc:1397] T a74b4d78d73a47d4a3d849053b2d454d P 43f12639d53d4d19b96ec67379f6fa1f: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.004s
I20260924 09:56:38.571995 8510 raft_consensus.cc:348] T a74b4d78d73a47d4a3d849053b2d454d P 43f12639d53d4d19b96ec67379f6fa1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e6632b5e12e420283aca499e3c94c42" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40363 } } peers { permanent_uuid: "43f12639d53d4d19b96ec67379f6fa1f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41195 } } peers { permanent_uuid: "818b4870af8045b9ac5ed6fdd83d89dd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36899 } }
I20260924 09:56:38.572319 8510 raft_consensus.cc:374] T a74b4d78d73a47d4a3d849053b2d454d P 43f12639d53d4d19b96ec67379f6fa1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 09:56:38.572446 8510 raft_consensus.cc:729] T a74b4d78d73a47d4a3d849053b2d454d P 43f12639d53d4d19b96ec67379f6fa1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43f12639d53d4d19b96ec67379f6fa1f, State: Initialized, Role: FOLLOWER
I20260924 09:56:38.572732 8510 consensus_queue.cc:260] T a74b4d78d73a47d4a3d849053b2d454d P 43f12639d53d4d19b96ec67379f6fa1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e6632b5e12e420283aca499e3c94c42" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40363 } } peers { permanent_uuid: "43f12639d53d4d19b96ec67379f6fa1f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41195 } } peers { permanent_uuid: "818b4870af8045b9ac5ed6fdd83d89dd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36899 } }
I20260924 09:56:38.573235 8529 ts_tablet_manager.cc:1918] T 52fb6058be404bb5ab25604fc2aba32c P 43f12639d53d4d19b96ec67379f6fa1f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 09:56:38.573371 8510 ts_tablet_manager.cc:1428] T a74b4d78d73a47d4a3d849053b2d454d P 43f12639d53d4d19b96ec67379f6fa1f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 09:56:38.573412 8529 log.cc:1196] T 52fb6058be404bb5ab25604fc2aba32c P 43f12639d53d4d19b96ec67379f6fa1f: Deleting WAL directory at /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790243735634835-388-0/minicluster-data/ts-1/wal/wals/52fb6058be404bb5ab25604fc2aba32c
I20260924 09:56:38.574101 8529 ts_tablet_manager.cc:1939] T 52fb6058be404bb5ab25604fc2aba32c P 43f12639d53d4d19b96ec67379f6fa1f: Deleting consensus metadata
I20260924 09:56:38.575788 7982 catalog_manager.cc:4841] TS 43f12639d53d4d19b96ec67379f6fa1f (127.0.97.2:41195): tablet 52fb6058be404bb5ab25604fc2aba32c (table test-table [id=99527f83296b4d7fbbbbf27eab316731]) successfully deleted
I20260924 09:56:38.575989 8506 tablet_bootstrap.cc:492] T a74b4d78d73a47d4a3d849053b2d454d P 818b4870af8045b9ac5ed6fdd83d89dd: No bootstrap required, opened a new log
I20260924 09:56:38.576125 8506 ts_tablet_manager.cc:1397] T a74b4d78d73a47d4a3d849053b2d454d P 818b4870af8045b9ac5ed6fdd83d89dd: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20260924 09:56:38.576746 8506 raft_consensus.cc:348] T a74b4d78d73a47d4a3d849053b2d454d P 818b4870af8045b9ac5ed6fdd83d89dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e6632b5e12e420283aca499e3c94c42" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40363 } } peers { permanent_uuid: "43f12639d53d4d19b96ec67379f6fa1f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41195 } } peers { permanent_uuid: "818b4870af8045b9ac5ed6fdd83d89dd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36899 } }
I20260924 09:56:38.577374 8506 raft_consensus.cc:374] T a74b4d78d73a47d4a3d849053b2d454d P 818b4870af8045b9ac5ed6fdd83d89dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 09:56:38.577535 8506 raft_consensus.cc:729] T a74b4d78d73a47d4a3d849053b2d454d P 818b4870af8045b9ac5ed6fdd83d89dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 818b4870af8045b9ac5ed6fdd83d89dd, State: Initialized, Role: FOLLOWER
I20260924 09:56:38.577809 8507 tablet_bootstrap.cc:492] T a74b4d78d73a47d4a3d849053b2d454d P 3e6632b5e12e420283aca499e3c94c42: No bootstrap required, opened a new log
I20260924 09:56:38.577980 8507 ts_tablet_manager.cc:1397] T a74b4d78d73a47d4a3d849053b2d454d P 3e6632b5e12e420283aca499e3c94c42: Time spent bootstrapping tablet: real 0.020s user 0.010s sys 0.000s
I20260924 09:56:38.577895 8506 consensus_queue.cc:260] T a74b4d78d73a47d4a3d849053b2d454d P 818b4870af8045b9ac5ed6fdd83d89dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e6632b5e12e420283aca499e3c94c42" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40363 } } peers { permanent_uuid: "43f12639d53d4d19b96ec67379f6fa1f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41195 } } peers { permanent_uuid: "818b4870af8045b9ac5ed6fdd83d89dd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36899 } }
I20260924 09:56:38.578513 8506 ts_tablet_manager.cc:1428] T a74b4d78d73a47d4a3d849053b2d454d P 818b4870af8045b9ac5ed6fdd83d89dd: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 09:56:38.578711 8507 raft_consensus.cc:348] T a74b4d78d73a47d4a3d849053b2d454d P 3e6632b5e12e420283aca499e3c94c42 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e6632b5e12e420283aca499e3c94c42" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40363 } } peers { permanent_uuid: "43f12639d53d4d19b96ec67379f6fa1f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41195 } } peers { permanent_uuid: "818b4870af8045b9ac5ed6fdd83d89dd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36899 } }
I20260924 09:56:38.578917 8507 raft_consensus.cc:374] T a74b4d78d73a47d4a3d849053b2d454d P 3e6632b5e12e420283aca499e3c94c42 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 09:56:38.579017 8507 raft_consensus.cc:729] T a74b4d78d73a47d4a3d849053b2d454d P 3e6632b5e12e420283aca499e3c94c42 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e6632b5e12e420283aca499e3c94c42, State: Initialized, Role: FOLLOWER
I20260924 09:56:38.579289 8507 consensus_queue.cc:260] T a74b4d78d73a47d4a3d849053b2d454d P 3e6632b5e12e420283aca499e3c94c42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e6632b5e12e420283aca499e3c94c42" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40363 } } peers { permanent_uuid: "43f12639d53d4d19b96ec67379f6fa1f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41195 } } peers { permanent_uuid: "818b4870af8045b9ac5ed6fdd83d89dd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36899 } }
I20260924 09:56:38.579825 8507 ts_tablet_manager.cc:1428] T a74b4d78d73a47d4a3d849053b2d454d P 3e6632b5e12e420283aca499e3c94c42: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 09:56:38.596189 8514 raft_consensus.cc:482] T a74b4d78d73a47d4a3d849053b2d454d P 43f12639d53d4d19b96ec67379f6fa1f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 09:56:38.596432 8514 raft_consensus.cc:504] T a74b4d78d73a47d4a3d849053b2d454d P 43f12639d53d4d19b96ec67379f6fa1f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e6632b5e12e420283aca499e3c94c42" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40363 } } peers { permanent_uuid: "43f12639d53d4d19b96ec67379f6fa1f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41195 } } peers { permanent_uuid: "818b4870af8045b9ac5ed6fdd83d89dd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36899 } }
I20260924 09:56:38.598232 8514 leader_election.cc:288] T a74b4d78d73a47d4a3d849053b2d454d P 43f12639d53d4d19b96ec67379f6fa1f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3e6632b5e12e420283aca499e3c94c42 (127.0.97.1:40363), 818b4870af8045b9ac5ed6fdd83d89dd (127.0.97.3:36899)
I20260924 09:56:38.619052 8501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:38.599204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:60265 (local address 127.0.97.1:40363)
0924 09:56:38.599394 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0924 09:56:38.599399 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 09:56:38.611763 (+ 12364us) server_negotiation.cc:407] Connection header received
0924 09:56:38.612157 (+ 394us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:56:38.612163 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:56:38.612258 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:56:38.612317 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 09:56:38.613114 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:38.614209 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:38.616704 (+ 2495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:38.617925 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:38.618014 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:38.618112 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0924 09:56:38.618668 (+ 556us) server_negotiation.cc:299] Negotiation successful
0924 09:56:38.618799 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260924 09:56:38.619762 8535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:38.598533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40363 (local address 127.0.97.2:60265)
0924 09:56:38.611662 (+ 13129us) negotiation.cc:107] Waiting for socket to connect
0924 09:56:38.611681 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 09:56:38.611997 (+ 316us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:56:38.612501 (+ 504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:56:38.612506 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:56:38.612547 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 09:56:38.612872 (+ 325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:38.612887 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:38.614373 (+ 1486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:38.614379 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:38.616540 (+ 2161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:38.616554 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:38.618304 (+ 1750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:38.618310 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:38.618495 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:38.618529 (+ 34us) client_negotiation.cc:769] Sending connection context
0924 09:56:38.619310 (+ 781us) client_negotiation.cc:240] Negotiation successful
0924 09:56:38.619458 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12998,"threads_started":1}
I20260924 09:56:38.620560 8154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a74b4d78d73a47d4a3d849053b2d454d" candidate_uuid: "43f12639d53d4d19b96ec67379f6fa1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e6632b5e12e420283aca499e3c94c42" is_pre_election: true
I20260924 09:56:38.620795 8154 raft_consensus.cc:2454] T a74b4d78d73a47d4a3d849053b2d454d P 3e6632b5e12e420283aca499e3c94c42 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 43f12639d53d4d19b96ec67379f6fa1f in term 0.
I20260924 09:56:38.621366 8225 leader_election.cc:302] T a74b4d78d73a47d4a3d849053b2d454d P 43f12639d53d4d19b96ec67379f6fa1f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e6632b5e12e420283aca499e3c94c42, 43f12639d53d4d19b96ec67379f6fa1f; no voters:
I20260924 09:56:38.621783 8514 raft_consensus.cc:2781] T a74b4d78d73a47d4a3d849053b2d454d P 43f12639d53d4d19b96ec67379f6fa1f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 09:56:38.621889 8514 raft_consensus.cc:482] T a74b4d78d73a47d4a3d849053b2d454d P 43f12639d53d4d19b96ec67379f6fa1f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 09:56:38.621961 8514 raft_consensus.cc:3037] T a74b4d78d73a47d4a3d849053b2d454d P 43f12639d53d4d19b96ec67379f6fa1f [term 0 FOLLOWER]: Advancing to term 1
I20260924 09:56:38.625151 8514 raft_consensus.cc:504] T a74b4d78d73a47d4a3d849053b2d454d P 43f12639d53d4d19b96ec67379f6fa1f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e6632b5e12e420283aca499e3c94c42" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40363 } } peers { permanent_uuid: "43f12639d53d4d19b96ec67379f6fa1f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41195 } } peers { permanent_uuid: "818b4870af8045b9ac5ed6fdd83d89dd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36899 } }
I20260924 09:56:38.626032 8514 leader_election.cc:288] T a74b4d78d73a47d4a3d849053b2d454d P 43f12639d53d4d19b96ec67379f6fa1f [CANDIDATE]: Term 1 election: Requested vote from peers 3e6632b5e12e420283aca499e3c94c42 (127.0.97.1:40363), 818b4870af8045b9ac5ed6fdd83d89dd (127.0.97.3:36899)
I20260924 09:56:38.626654 8154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a74b4d78d73a47d4a3d849053b2d454d" candidate_uuid: "43f12639d53d4d19b96ec67379f6fa1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e6632b5e12e420283aca499e3c94c42"
I20260924 09:56:38.626847 8154 raft_consensus.cc:3037] T a74b4d78d73a47d4a3d849053b2d454d P 3e6632b5e12e420283aca499e3c94c42 [term 0 FOLLOWER]: Advancing to term 1
I20260924 09:56:38.629653 8154 raft_consensus.cc:2454] T a74b4d78d73a47d4a3d849053b2d454d P 3e6632b5e12e420283aca499e3c94c42 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43f12639d53d4d19b96ec67379f6fa1f in term 1.
I20260924 09:56:38.630436 8225 leader_election.cc:302] T a74b4d78d73a47d4a3d849053b2d454d P 43f12639d53d4d19b96ec67379f6fa1f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e6632b5e12e420283aca499e3c94c42, 43f12639d53d4d19b96ec67379f6fa1f; no voters:
I20260924 09:56:38.630716 8514 raft_consensus.cc:2781] T a74b4d78d73a47d4a3d849053b2d454d P 43f12639d53d4d19b96ec67379f6fa1f [term 1 FOLLOWER]: Leader election won for term 1
I20260924 09:56:38.632123 8504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:38.598906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:37915 (local address 127.0.97.3:36899)
0924 09:56:38.599062 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0924 09:56:38.599068 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 09:56:38.619738 (+ 20670us) server_negotiation.cc:407] Connection header received
0924 09:56:38.623997 (+ 4259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:56:38.624002 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:56:38.624095 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:56:38.624155 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 09:56:38.624895 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:38.625935 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:38.629570 (+ 3635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:38.630763 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:38.630857 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:38.630951 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0924 09:56:38.631645 (+ 694us) server_negotiation.cc:299] Negotiation successful
0924 09:56:38.631810 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 09:56:38.632890 8534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:38.598606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:36899 (local address 127.0.97.2:37915)
0924 09:56:38.619630 (+ 21024us) negotiation.cc:107] Waiting for socket to connect
0924 09:56:38.619649 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 09:56:38.623809 (+ 4160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:56:38.624330 (+ 521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:56:38.624335 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:56:38.624357 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 09:56:38.624718 (+ 361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:38.624730 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:38.626200 (+ 1470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:38.626205 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:38.629380 (+ 3175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:38.629393 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:38.631222 (+ 1829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:38.631228 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:38.631457 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:38.631490 (+ 33us) client_negotiation.cc:769] Sending connection context
0924 09:56:38.632378 (+ 888us) client_negotiation.cc:240] Negotiation successful
0924 09:56:38.632529 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20883,"thread_start_us":13160,"threads_started":1}
I20260924 09:56:38.633925 8514 raft_consensus.cc:686] T a74b4d78d73a47d4a3d849053b2d454d P 43f12639d53d4d19b96ec67379f6fa1f [term 1 LEADER]: Becoming Leader. State: Replica: 43f12639d53d4d19b96ec67379f6fa1f, State: Running, Role: LEADER
I20260924 09:56:38.633952 8437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a74b4d78d73a47d4a3d849053b2d454d" candidate_uuid: "43f12639d53d4d19b96ec67379f6fa1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "818b4870af8045b9ac5ed6fdd83d89dd" is_pre_election: true
I20260924 09:56:38.634306 8437 raft_consensus.cc:2454] T a74b4d78d73a47d4a3d849053b2d454d P 818b4870af8045b9ac5ed6fdd83d89dd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 43f12639d53d4d19b96ec67379f6fa1f in term 0.
I20260924 09:56:38.634375 8514 consensus_queue.cc:237] T a74b4d78d73a47d4a3d849053b2d454d P 43f12639d53d4d19b96ec67379f6fa1f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e6632b5e12e420283aca499e3c94c42" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40363 } } peers { permanent_uuid: "43f12639d53d4d19b96ec67379f6fa1f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41195 } } peers { permanent_uuid: "818b4870af8045b9ac5ed6fdd83d89dd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36899 } }
I20260924 09:56:38.634680 8435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a74b4d78d73a47d4a3d849053b2d454d" candidate_uuid: "43f12639d53d4d19b96ec67379f6fa1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "818b4870af8045b9ac5ed6fdd83d89dd"
I20260924 09:56:38.634932 8435 raft_consensus.cc:3037] T a74b4d78d73a47d4a3d849053b2d454d P 818b4870af8045b9ac5ed6fdd83d89dd [term 0 FOLLOWER]: Advancing to term 1
I20260924 09:56:38.638165 8435 raft_consensus.cc:2454] T a74b4d78d73a47d4a3d849053b2d454d P 818b4870af8045b9ac5ed6fdd83d89dd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43f12639d53d4d19b96ec67379f6fa1f in term 1.
I20260924 09:56:38.638746 7998 catalog_manager.cc:5495] T a74b4d78d73a47d4a3d849053b2d454d P 43f12639d53d4d19b96ec67379f6fa1f reported cstate change: term changed from 0 to 1, leader changed from <none> to 43f12639d53d4d19b96ec67379f6fa1f (127.0.97.2). New cstate: current_term: 1 leader_uuid: "43f12639d53d4d19b96ec67379f6fa1f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e6632b5e12e420283aca499e3c94c42" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40363 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "43f12639d53d4d19b96ec67379f6fa1f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41195 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "818b4870af8045b9ac5ed6fdd83d89dd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36899 } health_report { overall_health: UNKNOWN } } }
Sep 24 09:56:38 dist-test-slave-w8d2 krb5kdc[7953](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243796, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260924 09:56:38.700958 8544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:38.677642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36706 (local address 127.0.97.62:38371)
0924 09:56:38.678651 (+ 1009us) server_negotiation.cc:206] Beginning negotiation
0924 09:56:38.678655 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 09:56:38.678674 (+ 19us) server_negotiation.cc:407] Connection header received
0924 09:56:38.678716 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:56:38.678719 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:56:38.678771 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:56:38.678853 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:56:38.680315 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:38.681138 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:38.682966 (+ 1828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:38.683400 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:38.683488 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:38.695468 (+ 11980us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:56:38.695474 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:56:38.695478 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:56:38.695547 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:56:38.698001 (+ 2454us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:56:38.698855 (+ 854us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:56:38.698859 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:56:38.698861 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:56:38.698999 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:56:38.699417 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:56:38.699420 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:56:38.699422 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:56:38.699703 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:56:38.699825 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0924 09:56:38.700344 (+ 519us) server_negotiation.cc:299] Negotiation successful
0924 09:56:38.700634 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":97,"server-negotiator.queue_time_us":922,"thread_start_us":739,"threads_started":1}
I20260924 09:56:38.700999 8543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:38.677623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38371 (local address 127.0.0.1:36706)
0924 09:56:38.678347 (+ 724us) negotiation.cc:107] Waiting for socket to connect
0924 09:56:38.678367 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 09:56:38.678517 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:56:38.678921 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:56:38.678929 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:56:38.679833 (+ 904us) client_negotiation.cc:189] Negotiated authn=SASL
0924 09:56:38.680136 (+ 303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:38.680148 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:38.681252 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:38.681258 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:38.682843 (+ 1585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:38.682856 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:38.683567 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:38.683573 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:38.683746 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:38.692281 (+ 8535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 09:56:38.692310 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0924 09:56:38.695170 (+ 2860us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 09:56:38.698134 (+ 2964us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 09:56:38.698150 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 09:56:38.698158 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0924 09:56:38.698721 (+ 563us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 09:56:38.699104 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 09:56:38.699110 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 09:56:38.699114 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 09:56:38.699317 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 09:56:38.699826 (+ 509us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 09:56:38.699835 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 09:56:38.700036 (+ 201us) client_negotiation.cc:769] Sending connection context
0924 09:56:38.700221 (+ 185us) client_negotiation.cc:240] Negotiation successful
0924 09:56:38.700515 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":534,"thread_start_us":361,"threads_started":1}
I20260924 09:56:38.706353 7998 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:36706:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260924 09:56:38.707376 7998 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 09:56:38.713553 8134 tablet_service.cc:1467] Processing CreateTablet for tablet 29dcb374ce0e4c7b9f7091f0a747a0b5 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c3a9d2cc8bf04047a011aeac59dbc95f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 09:56:38.713699 8277 tablet_service.cc:1467] Processing CreateTablet for tablet 29dcb374ce0e4c7b9f7091f0a747a0b5 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c3a9d2cc8bf04047a011aeac59dbc95f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 09:56:38.713945 8412 tablet_service.cc:1467] Processing CreateTablet for tablet 29dcb374ce0e4c7b9f7091f0a747a0b5 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c3a9d2cc8bf04047a011aeac59dbc95f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 09:56:38.714164 8134 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29dcb374ce0e4c7b9f7091f0a747a0b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 09:56:38.714196 8277 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29dcb374ce0e4c7b9f7091f0a747a0b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 09:56:38.714419 8412 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29dcb374ce0e4c7b9f7091f0a747a0b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 09:56:38.720774 8510 tablet_bootstrap.cc:492] T 29dcb374ce0e4c7b9f7091f0a747a0b5 P 43f12639d53d4d19b96ec67379f6fa1f: Bootstrap starting.
I20260924 09:56:38.720872 8507 tablet_bootstrap.cc:492] T 29dcb374ce0e4c7b9f7091f0a747a0b5 P 3e6632b5e12e420283aca499e3c94c42: Bootstrap starting.
I20260924 09:56:38.722312 8510 tablet_bootstrap.cc:654] T 29dcb374ce0e4c7b9f7091f0a747a0b5 P 43f12639d53d4d19b96ec67379f6fa1f: Neither blocks nor log segments found. Creating new log.
I20260924 09:56:38.722312 8507 tablet_bootstrap.cc:654] T 29dcb374ce0e4c7b9f7091f0a747a0b5 P 3e6632b5e12e420283aca499e3c94c42: Neither blocks nor log segments found. Creating new log.
I20260924 09:56:38.724457 8510 tablet_bootstrap.cc:492] T 29dcb374ce0e4c7b9f7091f0a747a0b5 P 43f12639d53d4d19b96ec67379f6fa1f: No bootstrap required, opened a new log
I20260924 09:56:38.724632 8510 ts_tablet_manager.cc:1397] T 29dcb374ce0e4c7b9f7091f0a747a0b5 P 43f12639d53d4d19b96ec67379f6fa1f: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260924 09:56:38.725272 8506 tablet_bootstrap.cc:492] T 29dcb374ce0e4c7b9f7091f0a747a0b5 P 818b4870af8045b9ac5ed6fdd83d89dd: Bootstrap starting.
I20260924 09:56:38.726984 8506 tablet_bootstrap.cc:654] T 29dcb374ce0e4c7b9f7091f0a747a0b5 P 818b4870af8045b9ac5ed6fdd83d89dd: Neither blocks nor log segments found. Creating new log.
I20260924 09:56:38.728756 8507 tablet_bootstrap.cc:492] T 29dcb374ce0e4c7b9f7091f0a747a0b5 P 3e6632b5e12e420283aca499e3c94c42: No bootstrap required, opened a new log
I20260924 09:56:38.728732 8510 raft_consensus.cc:348] T 29dcb374ce0e4c7b9f7091f0a747a0b5 P 43f12639d53d4d19b96ec67379f6fa1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e6632b5e12e420283aca499e3c94c42" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40363 } } peers { permanent_uuid: "43f12639d53d4d19b96ec67379f6fa1f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41195 } } peers { permanent_uuid: "818b4870af8045b9ac5ed6fdd83d89dd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36899 } }
I20260924 09:56:38.728904 8507 ts_tablet_manager.cc:1397] T 29dcb374ce0e4c7b9f7091f0a747a0b5 P 3e6632b5e12e420283aca499e3c94c42: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260924 09:56:38.728961 8510 raft_consensus.cc:374] T 29dcb374ce0e4c7b9f7091f0a747a0b5 P 43f12639d53d4d19b96ec67379f6fa1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 09:56:38.729046 8510 raft_consensus.cc:729] T 29dcb374ce0e4c7b9f7091f0a747a0b5 P 43f12639d53d4d19b96ec67379f6fa1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43f12639d53d4d19b96ec67379f6fa1f, State: Initialized, Role: FOLLOWER
I20260924 09:56:38.729310 8510 consensus_queue.cc:260] T 29dcb374ce0e4c7b9f7091f0a747a0b5 P 43f12639d53d4d19b96ec67379f6fa1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e6632b5e12e420283aca499e3c94c42" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40363 } } peers { permanent_uuid: "43f12639d53d4d19b96ec67379f6fa1f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41195 } } peers { permanent_uuid: "818b4870af8045b9ac5ed6fdd83d89dd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36899 } }
I20260924 09:56:38.729893 8510 ts_tablet_manager.cc:1428] T 29dcb374ce0e4c7b9f7091f0a747a0b5 P 43f12639d53d4d19b96ec67379f6fa1f: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20260924 09:56:38.729746 8514 tablet_replica.cc:439] T 29dcb374ce0e4c7b9f7091f0a747a0b5 P 43f12639d53d4d19b96ec67379f6fa1f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e6632b5e12e420283aca499e3c94c42" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40363 } } peers { permanent_uuid: "43f12639d53d4d19b96ec67379f6fa1f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41195 } } peers { permanent_uuid: "818b4870af8045b9ac5ed6fdd83d89dd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36899 } } }
I20260924 09:56:38.730052 8514 tablet_replica.cc:442] T 29dcb374ce0e4c7b9f7091f0a747a0b5 P 43f12639d53d4d19b96ec67379f6fa1f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 09:56:38.729578 8507 raft_consensus.cc:348] T 29dcb374ce0e4c7b9f7091f0a747a0b5 P 3e6632b5e12e420283aca499e3c94c42 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e6632b5e12e420283aca499e3c94c42" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40363 } } peers { permanent_uuid: "43f12639d53d4d19b96ec67379f6fa1f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41195 } } peers { permanent_uuid: "818b4870af8045b9ac5ed6fdd83d89dd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36899 } }
I20260924 09:56:38.730329 8507 raft_consensus.cc:374] T 29dcb374ce0e4c7b9f7091f0a747a0b5 P 3e6632b5e12e420283aca499e3c94c42 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 09:56:38.730409 8507 raft_consensus.cc:729] T 29dcb374ce0e4c7b9f7091f0a747a0b5 P 3e6632b5e12e420283aca499e3c94c42 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e6632b5e12e420283aca499e3c94c42, State: Initialized, Role: FOLLOWER
I20260924 09:56:38.730619 8507 consensus_queue.cc:260] T 29dcb374ce0e4c7b9f7091f0a747a0b5 P 3e6632b5e12e420283aca499e3c94c42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e6632b5e12e420283aca499e3c94c42" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40363 } } peers { permanent_uuid: "43f12639d53d4d19b96ec67379f6fa1f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41195 } } peers { permanent_uuid: "818b4870af8045b9ac5ed6fdd83d89dd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36899 } }
I20260924 09:56:38.731022 8513 tablet_replica.cc:439] T 29dcb374ce0e4c7b9f7091f0a747a0b5 P 3e6632b5e12e420283aca499e3c94c42: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e6632b5e12e420283aca499e3c94c42" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40363 } } peers { permanent_uuid: "43f12639d53d4d19b96ec67379f6fa1f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41195 } } peers { permanent_uuid: "818b4870af8045b9ac5ed6fdd83d89dd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36899 } } }
I20260924 09:56:38.732234 8513 tablet_replica.cc:442] T 29dcb374ce0e4c7b9f7091f0a747a0b5 P 3e6632b5e12e420283aca499e3c94c42: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 09:56:38.732602 8506 tablet_bootstrap.cc:492] T 29dcb374ce0e4c7b9f7091f0a747a0b5 P 818b4870af8045b9ac5ed6fdd83d89dd: No bootstrap required, opened a new log
I20260924 09:56:38.732746 8506 ts_tablet_manager.cc:1397] T 29dcb374ce0e4c7b9f7091f0a747a0b5 P 818b4870af8045b9ac5ed6fdd83d89dd: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.003s
I20260924 09:56:38.733559 8506 raft_consensus.cc:348] T 29dcb374ce0e4c7b9f7091f0a747a0b5 P 818b4870af8045b9ac5ed6fdd83d89dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e6632b5e12e420283aca499e3c94c42" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40363 } } peers { permanent_uuid: "43f12639d53d4d19b96ec67379f6fa1f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41195 } } peers { permanent_uuid: "818b4870af8045b9ac5ed6fdd83d89dd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36899 } }
I20260924 09:56:38.733844 8506 raft_consensus.cc:374] T 29dcb374ce0e4c7b9f7091f0a747a0b5 P 818b4870af8045b9ac5ed6fdd83d89dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 09:56:38.733964 8506 raft_consensus.cc:729] T 29dcb374ce0e4c7b9f7091f0a747a0b5 P 818b4870af8045b9ac5ed6fdd83d89dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 818b4870af8045b9ac5ed6fdd83d89dd, State: Initialized, Role: FOLLOWER
I20260924 09:56:38.734243 8506 consensus_queue.cc:260] T 29dcb374ce0e4c7b9f7091f0a747a0b5 P 818b4870af8045b9ac5ed6fdd83d89dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e6632b5e12e420283aca499e3c94c42" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40363 } } peers { permanent_uuid: "43f12639d53d4d19b96ec67379f6fa1f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41195 } } peers { permanent_uuid: "818b4870af8045b9ac5ed6fdd83d89dd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36899 } }
I20260924 09:56:38.734489 8514 raft_consensus.cc:482] T 29dcb374ce0e4c7b9f7091f0a747a0b5 P 43f12639d53d4d19b96ec67379f6fa1f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 09:56:38.734650 8514 raft_consensus.cc:504] T 29dcb374ce0e4c7b9f7091f0a747a0b5 P 43f12639d53d4d19b96ec67379f6fa1f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e6632b5e12e420283aca499e3c94c42" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40363 } } peers { permanent_uuid: "43f12639d53d4d19b96ec67379f6fa1f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41195 } } peers { permanent_uuid: "818b4870af8045b9ac5ed6fdd83d89dd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36899 } }
I20260924 09:56:38.734761 8507 ts_tablet_manager.cc:1428] T 29dcb374ce0e4c7b9f7091f0a747a0b5 P 3e6632b5e12e420283aca499e3c94c42: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20260924 09:56:38.734934 8506 ts_tablet_manager.cc:1428] T 29dcb374ce0e4c7b9f7091f0a747a0b5 P 818b4870af8045b9ac5ed6fdd83d89dd: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260924 09:56:38.735427 8512 tablet_replica.cc:439] T 29dcb374ce0e4c7b9f7091f0a747a0b5 P 818b4870af8045b9ac5ed6fdd83d89dd: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e6632b5e12e420283aca499e3c94c42" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40363 } } peers { permanent_uuid: "43f12639d53d4d19b96ec67379f6fa1f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41195 } } peers { permanent_uuid: "818b4870af8045b9ac5ed6fdd83d89dd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36899 } } }
I20260924 09:56:38.735680 8512 tablet_replica.cc:442] T 29dcb374ce0e4c7b9f7091f0a747a0b5 P 818b4870af8045b9ac5ed6fdd83d89dd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 09:56:38.735968 8154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29dcb374ce0e4c7b9f7091f0a747a0b5" candidate_uuid: "43f12639d53d4d19b96ec67379f6fa1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e6632b5e12e420283aca499e3c94c42" is_pre_election: true
I20260924 09:56:38.736188 8154 raft_consensus.cc:2454] T 29dcb374ce0e4c7b9f7091f0a747a0b5 P 3e6632b5e12e420283aca499e3c94c42 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 43f12639d53d4d19b96ec67379f6fa1f in term 0.
I20260924 09:56:38.736564 8225 leader_election.cc:302] T 29dcb374ce0e4c7b9f7091f0a747a0b5 P 43f12639d53d4d19b96ec67379f6fa1f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e6632b5e12e420283aca499e3c94c42, 43f12639d53d4d19b96ec67379f6fa1f; no voters:
I20260924 09:56:38.736821 8536 raft_consensus.cc:2781] T 29dcb374ce0e4c7b9f7091f0a747a0b5 P 43f12639d53d4d19b96ec67379f6fa1f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 09:56:38.736788 8435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29dcb374ce0e4c7b9f7091f0a747a0b5" candidate_uuid: "43f12639d53d4d19b96ec67379f6fa1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "818b4870af8045b9ac5ed6fdd83d89dd" is_pre_election: true
I20260924 09:56:38.736919 8536 raft_consensus.cc:482] T 29dcb374ce0e4c7b9f7091f0a747a0b5 P 43f12639d53d4d19b96ec67379f6fa1f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 09:56:38.737031 8536 raft_consensus.cc:3037] T 29dcb374ce0e4c7b9f7091f0a747a0b5 P 43f12639d53d4d19b96ec67379f6fa1f [term 0 FOLLOWER]: Advancing to term 1
I20260924 09:56:38.737043 8435 raft_consensus.cc:2454] T 29dcb374ce0e4c7b9f7091f0a747a0b5 P 818b4870af8045b9ac5ed6fdd83d89dd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 43f12639d53d4d19b96ec67379f6fa1f in term 0.
I20260924 09:56:38.738948 8514 leader_election.cc:288] T 29dcb374ce0e4c7b9f7091f0a747a0b5 P 43f12639d53d4d19b96ec67379f6fa1f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3e6632b5e12e420283aca499e3c94c42 (127.0.97.1:40363), 818b4870af8045b9ac5ed6fdd83d89dd (127.0.97.3:36899)
I20260924 09:56:38.739876 8536 raft_consensus.cc:504] T 29dcb374ce0e4c7b9f7091f0a747a0b5 P 43f12639d53d4d19b96ec67379f6fa1f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e6632b5e12e420283aca499e3c94c42" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40363 } } peers { permanent_uuid: "43f12639d53d4d19b96ec67379f6fa1f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41195 } } peers { permanent_uuid: "818b4870af8045b9ac5ed6fdd83d89dd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36899 } }
I20260924 09:56:38.740399 8536 leader_election.cc:288] T 29dcb374ce0e4c7b9f7091f0a747a0b5 P 43f12639d53d4d19b96ec67379f6fa1f [CANDIDATE]: Term 1 election: Requested vote from peers 3e6632b5e12e420283aca499e3c94c42 (127.0.97.1:40363), 818b4870af8045b9ac5ed6fdd83d89dd (127.0.97.3:36899)
I20260924 09:56:38.740726 8154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29dcb374ce0e4c7b9f7091f0a747a0b5" candidate_uuid: "43f12639d53d4d19b96ec67379f6fa1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e6632b5e12e420283aca499e3c94c42"
I20260924 09:56:38.740834 8435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29dcb374ce0e4c7b9f7091f0a747a0b5" candidate_uuid: "43f12639d53d4d19b96ec67379f6fa1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "818b4870af8045b9ac5ed6fdd83d89dd"
I20260924 09:56:38.740913 8154 raft_consensus.cc:3037] T 29dcb374ce0e4c7b9f7091f0a747a0b5 P 3e6632b5e12e420283aca499e3c94c42 [term 0 FOLLOWER]: Advancing to term 1
I20260924 09:56:38.741044 8435 raft_consensus.cc:3037] T 29dcb374ce0e4c7b9f7091f0a747a0b5 P 818b4870af8045b9ac5ed6fdd83d89dd [term 0 FOLLOWER]: Advancing to term 1
I20260924 09:56:38.743572 8154 raft_consensus.cc:2454] T 29dcb374ce0e4c7b9f7091f0a747a0b5 P 3e6632b5e12e420283aca499e3c94c42 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43f12639d53d4d19b96ec67379f6fa1f in term 1.
I20260924 09:56:38.743916 8225 leader_election.cc:302] T 29dcb374ce0e4c7b9f7091f0a747a0b5 P 43f12639d53d4d19b96ec67379f6fa1f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e6632b5e12e420283aca499e3c94c42, 43f12639d53d4d19b96ec67379f6fa1f; no voters:
I20260924 09:56:38.744060 8435 raft_consensus.cc:2454] T 29dcb374ce0e4c7b9f7091f0a747a0b5 P 818b4870af8045b9ac5ed6fdd83d89dd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43f12639d53d4d19b96ec67379f6fa1f in term 1.
I20260924 09:56:38.745920 8536 raft_consensus.cc:2781] T 29dcb374ce0e4c7b9f7091f0a747a0b5 P 43f12639d53d4d19b96ec67379f6fa1f [term 1 FOLLOWER]: Leader election won for term 1
I20260924 09:56:38.746114 8536 raft_consensus.cc:686] T 29dcb374ce0e4c7b9f7091f0a747a0b5 P 43f12639d53d4d19b96ec67379f6fa1f [term 1 LEADER]: Becoming Leader. State: Replica: 43f12639d53d4d19b96ec67379f6fa1f, State: Running, Role: LEADER
I20260924 09:56:38.746410 8536 consensus_queue.cc:237] T 29dcb374ce0e4c7b9f7091f0a747a0b5 P 43f12639d53d4d19b96ec67379f6fa1f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e6632b5e12e420283aca499e3c94c42" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40363 } } peers { permanent_uuid: "43f12639d53d4d19b96ec67379f6fa1f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41195 } } peers { permanent_uuid: "818b4870af8045b9ac5ed6fdd83d89dd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36899 } }
I20260924 09:56:38.748471 8514 tablet_replica.cc:439] T 29dcb374ce0e4c7b9f7091f0a747a0b5 P 43f12639d53d4d19b96ec67379f6fa1f: TxnStatusTablet state changed. Reason: New leader 43f12639d53d4d19b96ec67379f6fa1f. Latest consensus state: current_term: 1 leader_uuid: "43f12639d53d4d19b96ec67379f6fa1f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e6632b5e12e420283aca499e3c94c42" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40363 } } peers { permanent_uuid: "43f12639d53d4d19b96ec67379f6fa1f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41195 } } peers { permanent_uuid: "818b4870af8045b9ac5ed6fdd83d89dd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36899 } } }
I20260924 09:56:38.748734 8514 tablet_replica.cc:442] T 29dcb374ce0e4c7b9f7091f0a747a0b5 P 43f12639d53d4d19b96ec67379f6fa1f: This TxnStatusTablet replica's current role is: LEADER
I20260924 09:56:38.749313 8549 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260924 09:56:38.749584 8549 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260924 09:56:38.750658 7997 catalog_manager.cc:5495] T 29dcb374ce0e4c7b9f7091f0a747a0b5 P 43f12639d53d4d19b96ec67379f6fa1f reported cstate change: term changed from 0 to 1, leader changed from <none> to 43f12639d53d4d19b96ec67379f6fa1f (127.0.97.2). New cstate: current_term: 1 leader_uuid: "43f12639d53d4d19b96ec67379f6fa1f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e6632b5e12e420283aca499e3c94c42" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40363 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "43f12639d53d4d19b96ec67379f6fa1f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41195 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "818b4870af8045b9ac5ed6fdd83d89dd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36899 } health_report { overall_health: UNKNOWN } } }
I20260924 09:56:38.766098 8505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:38.760871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36832 (local address 127.0.97.2:41195)
0924 09:56:38.761020 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0924 09:56:38.761026 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 09:56:38.761232 (+ 206us) server_negotiation.cc:407] Connection header received
0924 09:56:38.761468 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:56:38.761474 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:56:38.761612 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:56:38.761724 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 09:56:38.762056 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:38.762785 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:38.764074 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:38.764485 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:38.764606 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:38.765010 (+ 404us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 09:56:38.765474 (+ 464us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 09:56:38.765601 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0924 09:56:38.765710 (+ 109us) server_negotiation.cc:299] Negotiation successful
0924 09:56:38.765790 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 09:56:38.766184 8543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:38.760841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:41195 (local address 127.0.0.1:36832)
0924 09:56:38.761155 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0924 09:56:38.761172 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 09:56:38.761296 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:56:38.761761 (+ 465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:56:38.761766 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:56:38.761778 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 09:56:38.761941 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:38.761950 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:38.762908 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:38.762913 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:38.763950 (+ 1037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:38.763961 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:38.764630 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:38.764636 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:38.764801 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:38.764851 (+ 50us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 09:56:38.765585 (+ 734us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 09:56:38.765602 (+ 17us) client_negotiation.cc:769] Sending connection context
0924 09:56:38.765677 (+ 75us) client_negotiation.cc:240] Negotiation successful
0924 09:56:38.765915 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147}
I20260924 09:56:38.769469 8154 raft_consensus.cc:1261] T 29dcb374ce0e4c7b9f7091f0a747a0b5 P 3e6632b5e12e420283aca499e3c94c42 [term 1 FOLLOWER]: Refusing update from remote peer 43f12639d53d4d19b96ec67379f6fa1f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 09:56:38.769767 8435 raft_consensus.cc:1261] T 29dcb374ce0e4c7b9f7091f0a747a0b5 P 818b4870af8045b9ac5ed6fdd83d89dd [term 1 FOLLOWER]: Refusing update from remote peer 43f12639d53d4d19b96ec67379f6fa1f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 09:56:38.770174 8536 consensus_queue.cc:1035] T 29dcb374ce0e4c7b9f7091f0a747a0b5 P 43f12639d53d4d19b96ec67379f6fa1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e6632b5e12e420283aca499e3c94c42" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40363 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 09:56:38.770476 8514 consensus_queue.cc:1035] T 29dcb374ce0e4c7b9f7091f0a747a0b5 P 43f12639d53d4d19b96ec67379f6fa1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "818b4870af8045b9ac5ed6fdd83d89dd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 09:56:38.774209 8512 tablet_replica.cc:439] T 29dcb374ce0e4c7b9f7091f0a747a0b5 P 818b4870af8045b9ac5ed6fdd83d89dd: TxnStatusTablet state changed. Reason: New leader 43f12639d53d4d19b96ec67379f6fa1f. Latest consensus state: current_term: 1 leader_uuid: "43f12639d53d4d19b96ec67379f6fa1f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e6632b5e12e420283aca499e3c94c42" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40363 } } peers { permanent_uuid: "43f12639d53d4d19b96ec67379f6fa1f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41195 } } peers { permanent_uuid: "818b4870af8045b9ac5ed6fdd83d89dd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36899 } } }
I20260924 09:56:38.774508 8512 tablet_replica.cc:442] T 29dcb374ce0e4c7b9f7091f0a747a0b5 P 818b4870af8045b9ac5ed6fdd83d89dd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 09:56:38.778425 8519 mvcc.cc:205] Tried to move back new op lower bound from 7332838599753830400 to 7332838599667900416. Current Snapshot: MvccSnapshot[applied={T|T < 7332838599753830400}]
I20260924 09:56:38.778697 8512 tablet_replica.cc:439] T 29dcb374ce0e4c7b9f7091f0a747a0b5 P 818b4870af8045b9ac5ed6fdd83d89dd: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "43f12639d53d4d19b96ec67379f6fa1f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e6632b5e12e420283aca499e3c94c42" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40363 } } peers { permanent_uuid: "43f12639d53d4d19b96ec67379f6fa1f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41195 } } peers { permanent_uuid: "818b4870af8045b9ac5ed6fdd83d89dd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36899 } } }
I20260924 09:56:38.778921 8512 tablet_replica.cc:442] T 29dcb374ce0e4c7b9f7091f0a747a0b5 P 818b4870af8045b9ac5ed6fdd83d89dd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 09:56:38.781279 8514 tablet_replica.cc:439] T 29dcb374ce0e4c7b9f7091f0a747a0b5 P 43f12639d53d4d19b96ec67379f6fa1f: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "43f12639d53d4d19b96ec67379f6fa1f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e6632b5e12e420283aca499e3c94c42" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40363 } } peers { permanent_uuid: "43f12639d53d4d19b96ec67379f6fa1f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41195 } } peers { permanent_uuid: "818b4870af8045b9ac5ed6fdd83d89dd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36899 } } }
I20260924 09:56:38.781517 8514 tablet_replica.cc:442] T 29dcb374ce0e4c7b9f7091f0a747a0b5 P 43f12639d53d4d19b96ec67379f6fa1f: This TxnStatusTablet replica's current role is: LEADER
I20260924 09:56:38.783331 8513 tablet_replica.cc:439] T 29dcb374ce0e4c7b9f7091f0a747a0b5 P 3e6632b5e12e420283aca499e3c94c42: TxnStatusTablet state changed. Reason: New leader 43f12639d53d4d19b96ec67379f6fa1f. Latest consensus state: current_term: 1 leader_uuid: "43f12639d53d4d19b96ec67379f6fa1f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e6632b5e12e420283aca499e3c94c42" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40363 } } peers { permanent_uuid: "43f12639d53d4d19b96ec67379f6fa1f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41195 } } peers { permanent_uuid: "818b4870af8045b9ac5ed6fdd83d89dd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36899 } } }
I20260924 09:56:38.783609 8513 tablet_replica.cc:442] T 29dcb374ce0e4c7b9f7091f0a747a0b5 P 3e6632b5e12e420283aca499e3c94c42: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 09:56:38.791718 8513 tablet_replica.cc:439] T 29dcb374ce0e4c7b9f7091f0a747a0b5 P 3e6632b5e12e420283aca499e3c94c42: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "43f12639d53d4d19b96ec67379f6fa1f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e6632b5e12e420283aca499e3c94c42" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40363 } } peers { permanent_uuid: "43f12639d53d4d19b96ec67379f6fa1f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41195 } } peers { permanent_uuid: "818b4870af8045b9ac5ed6fdd83d89dd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36899 } } }
I20260924 09:56:38.791929 8513 tablet_replica.cc:442] T 29dcb374ce0e4c7b9f7091f0a747a0b5 P 3e6632b5e12e420283aca499e3c94c42: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 09:56:38.792398 8505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:38.785585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36840 (local address 127.0.97.2:41195)
0924 09:56:38.785752 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0924 09:56:38.785757 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 09:56:38.785773 (+ 16us) server_negotiation.cc:407] Connection header received
0924 09:56:38.785831 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:56:38.785836 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:56:38.785907 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:56:38.785978 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 09:56:38.786525 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:38.787915 (+ 1390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:38.789919 (+ 2004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:38.790507 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:38.790612 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:38.791228 (+ 616us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 09:56:38.791366 (+ 138us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 09:56:38.791443 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 09:56:38.792054 (+ 611us) server_negotiation.cc:299] Negotiation successful
0924 09:56:38.792139 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 09:56:38.795423 8554 tablet_replica.cc:439] T 29dcb374ce0e4c7b9f7091f0a747a0b5 P 43f12639d53d4d19b96ec67379f6fa1f: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "43f12639d53d4d19b96ec67379f6fa1f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e6632b5e12e420283aca499e3c94c42" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40363 } } peers { permanent_uuid: "43f12639d53d4d19b96ec67379f6fa1f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41195 } } peers { permanent_uuid: "818b4870af8045b9ac5ed6fdd83d89dd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36899 } } }
I20260924 09:56:38.795897 8554 tablet_replica.cc:442] T 29dcb374ce0e4c7b9f7091f0a747a0b5 P 43f12639d53d4d19b96ec67379f6fa1f: This TxnStatusTablet replica's current role is: LEADER
I20260924 09:56:38.826192 8505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:38.808850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:46315 (local address 127.0.97.2:41195)
0924 09:56:38.809056 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0924 09:56:38.809062 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 09:56:38.819790 (+ 10728us) server_negotiation.cc:407] Connection header received
0924 09:56:38.819930 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:56:38.819935 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:56:38.820028 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:56:38.820135 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 09:56:38.820744 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:38.821869 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:38.823988 (+ 2119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:38.824671 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:38.824789 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:38.825371 (+ 582us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 09:56:38.825520 (+ 149us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 09:56:38.825607 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0924 09:56:38.825764 (+ 157us) server_negotiation.cc:299] Negotiation successful
0924 09:56:38.825846 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":27,"server-negotiator.queue_time_us":72}
I20260924 09:56:38.826280 8557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:38.808433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:41195 (local address 127.0.97.2:46315)
0924 09:56:38.819668 (+ 11235us) negotiation.cc:107] Waiting for socket to connect
0924 09:56:38.819685 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 09:56:38.819833 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:56:38.820171 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:56:38.820176 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:56:38.820188 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 09:56:38.820573 (+ 385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:38.820584 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:38.822010 (+ 1426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:38.822015 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:38.823839 (+ 1824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:38.823853 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:38.824876 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:38.824883 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:38.825054 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:38.825126 (+ 72us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 09:56:38.825633 (+ 507us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 09:56:38.825645 (+ 12us) client_negotiation.cc:769] Sending connection context
0924 09:56:38.825735 (+ 90us) client_negotiation.cc:240] Negotiation successful
0924 09:56:38.825868 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11070,"thread_start_us":10936,"threads_started":1}
I20260924 09:56:38.837706 8435 raft_consensus.cc:1261] T a74b4d78d73a47d4a3d849053b2d454d P 818b4870af8045b9ac5ed6fdd83d89dd [term 1 FOLLOWER]: Refusing update from remote peer 43f12639d53d4d19b96ec67379f6fa1f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 09:56:38.837884 8154 raft_consensus.cc:1261] T a74b4d78d73a47d4a3d849053b2d454d P 3e6632b5e12e420283aca499e3c94c42 [term 1 FOLLOWER]: Refusing update from remote peer 43f12639d53d4d19b96ec67379f6fa1f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 09:56:38.838438 8536 consensus_queue.cc:1035] T a74b4d78d73a47d4a3d849053b2d454d P 43f12639d53d4d19b96ec67379f6fa1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "818b4870af8045b9ac5ed6fdd83d89dd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 09:56:38.838809 8514 consensus_queue.cc:1035] T a74b4d78d73a47d4a3d849053b2d454d P 43f12639d53d4d19b96ec67379f6fa1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e6632b5e12e420283aca499e3c94c42" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40363 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 09:56:38.852006 8520 mvcc.cc:205] Tried to move back new op lower bound from 7332838600034279424 to 7332838599211159552. Current Snapshot: MvccSnapshot[applied={T|T < 7332838600034279424}]
I20260924 09:56:38.855720 8521 mvcc.cc:205] Tried to move back new op lower bound from 7332838600034279424 to 7332838599211159552. Current Snapshot: MvccSnapshot[applied={T|T < 7332838600034279424}]
I20260924 09:56:38.923058 7998 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36702:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 09:56:38.923362 7998 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36702:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 09:56:38.926895 7998 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e3f28e3acf1e44e59f3ed21ee9c9bd10: Sending DeleteTablet for 3 replicas of tablet a74b4d78d73a47d4a3d849053b2d454d
I20260924 09:56:38.928125 8134 tablet_service.cc:1514] Processing DeleteTablet for tablet a74b4d78d73a47d4a3d849053b2d454d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 09:56:38 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:38830
I20260924 09:56:38.928419 8528 tablet_replica.cc:330] T a74b4d78d73a47d4a3d849053b2d454d P 3e6632b5e12e420283aca499e3c94c42: stopping tablet replica
I20260924 09:56:38.928614 8528 raft_consensus.cc:2229] T a74b4d78d73a47d4a3d849053b2d454d P 3e6632b5e12e420283aca499e3c94c42 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 09:56:38.928666 8412 tablet_service.cc:1514] Processing DeleteTablet for tablet a74b4d78d73a47d4a3d849053b2d454d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 09:56:38 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:50590
I20260924 09:56:38.928781 8528 raft_consensus.cc:2258] T a74b4d78d73a47d4a3d849053b2d454d P 3e6632b5e12e420283aca499e3c94c42 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 09:56:38.928903 8530 tablet_replica.cc:330] T a74b4d78d73a47d4a3d849053b2d454d P 818b4870af8045b9ac5ed6fdd83d89dd: stopping tablet replica
I20260924 09:56:38.929056 8530 raft_consensus.cc:2229] T a74b4d78d73a47d4a3d849053b2d454d P 818b4870af8045b9ac5ed6fdd83d89dd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 09:56:38.929241 8530 raft_consensus.cc:2258] T a74b4d78d73a47d4a3d849053b2d454d P 818b4870af8045b9ac5ed6fdd83d89dd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 09:56:38.929842 8528 ts_tablet_manager.cc:1905] T a74b4d78d73a47d4a3d849053b2d454d P 3e6632b5e12e420283aca499e3c94c42: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 09:56:38.930159 8277 tablet_service.cc:1514] Processing DeleteTablet for tablet a74b4d78d73a47d4a3d849053b2d454d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 09:56:38 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:36820
I20260924 09:56:38.930402 8529 tablet_replica.cc:330] T a74b4d78d73a47d4a3d849053b2d454d P 43f12639d53d4d19b96ec67379f6fa1f: stopping tablet replica
I20260924 09:56:38.930590 8529 raft_consensus.cc:2229] T a74b4d78d73a47d4a3d849053b2d454d P 43f12639d53d4d19b96ec67379f6fa1f [term 1 LEADER]: Raft consensus shutting down.
I20260924 09:56:38.931064 8529 raft_consensus.cc:2258] T a74b4d78d73a47d4a3d849053b2d454d P 43f12639d53d4d19b96ec67379f6fa1f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 09:56:38.932375 8529 ts_tablet_manager.cc:1905] T a74b4d78d73a47d4a3d849053b2d454d P 43f12639d53d4d19b96ec67379f6fa1f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 09:56:38.937115 8528 ts_tablet_manager.cc:1918] T a74b4d78d73a47d4a3d849053b2d454d P 3e6632b5e12e420283aca499e3c94c42: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 09:56:38.937279 8528 log.cc:1196] T a74b4d78d73a47d4a3d849053b2d454d P 3e6632b5e12e420283aca499e3c94c42: Deleting WAL directory at /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790243735634835-388-0/minicluster-data/ts-0/wal/wals/a74b4d78d73a47d4a3d849053b2d454d
I20260924 09:56:38.937673 8528 ts_tablet_manager.cc:1939] T a74b4d78d73a47d4a3d849053b2d454d P 3e6632b5e12e420283aca499e3c94c42: Deleting consensus metadata
I20260924 09:56:38.938962 7983 catalog_manager.cc:4841] TS 3e6632b5e12e420283aca499e3c94c42 (127.0.97.1:40363): tablet a74b4d78d73a47d4a3d849053b2d454d (table test-table [id=8c55507b7cde4cffbfa98ab00e5bec29]) successfully deleted
I20260924 09:56:38.939975 8530 ts_tablet_manager.cc:1905] T a74b4d78d73a47d4a3d849053b2d454d P 818b4870af8045b9ac5ed6fdd83d89dd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 09:56:38.945878 8529 ts_tablet_manager.cc:1918] T a74b4d78d73a47d4a3d849053b2d454d P 43f12639d53d4d19b96ec67379f6fa1f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 09:56:38.946125 8529 log.cc:1196] T a74b4d78d73a47d4a3d849053b2d454d P 43f12639d53d4d19b96ec67379f6fa1f: Deleting WAL directory at /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790243735634835-388-0/minicluster-data/ts-1/wal/wals/a74b4d78d73a47d4a3d849053b2d454d
I20260924 09:56:38.946597 8529 ts_tablet_manager.cc:1939] T a74b4d78d73a47d4a3d849053b2d454d P 43f12639d53d4d19b96ec67379f6fa1f: Deleting consensus metadata
I20260924 09:56:38.946980 8530 ts_tablet_manager.cc:1918] T a74b4d78d73a47d4a3d849053b2d454d P 818b4870af8045b9ac5ed6fdd83d89dd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 09:56:38.947144 8530 log.cc:1196] T a74b4d78d73a47d4a3d849053b2d454d P 818b4870af8045b9ac5ed6fdd83d89dd: Deleting WAL directory at /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790243735634835-388-0/minicluster-data/ts-2/wal/wals/a74b4d78d73a47d4a3d849053b2d454d
I20260924 09:56:38.947676 8530 ts_tablet_manager.cc:1939] T a74b4d78d73a47d4a3d849053b2d454d P 818b4870af8045b9ac5ed6fdd83d89dd: Deleting consensus metadata
I20260924 09:56:38.948768 7982 catalog_manager.cc:4841] TS 43f12639d53d4d19b96ec67379f6fa1f (127.0.97.2:41195): tablet a74b4d78d73a47d4a3d849053b2d454d (table test-table [id=8c55507b7cde4cffbfa98ab00e5bec29]) successfully deleted
I20260924 09:56:38.948765 7985 catalog_manager.cc:4841] TS 818b4870af8045b9ac5ed6fdd83d89dd (127.0.97.3:36899): tablet a74b4d78d73a47d4a3d849053b2d454d (table test-table [id=8c55507b7cde4cffbfa98ab00e5bec29]) successfully deleted
Sep 24 09:56:38 dist-test-slave-w8d2 krb5kdc[7953](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243798, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 09:56:38 dist-test-slave-w8d2 krb5kdc[7953](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 09:56:38 dist-test-slave-w8d2 krb5kdc[7953](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260924 09:56:39.003752 8544 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 09:56:38.991541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36708 (local address 127.0.97.62:38371)
0924 09:56:38.991735 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0924 09:56:38.991741 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 09:56:38.992238 (+ 497us) server_negotiation.cc:407] Connection header received
0924 09:56:38.992402 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:56:38.992408 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:56:38.992493 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:56:38.992606 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:56:38.994219 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:38.995375 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:38.997145 (+ 1770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:38.997815 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:38.997929 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:39.003490 (+ 5561us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36708: BlockingRecv error: recv got EOF from 127.0.0.1:36708 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Sep 24 09:56:39 dist-test-slave-w8d2 krb5kdc[7953](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243798, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260924 09:56:39.024085 8544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:39.007267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36722 (local address 127.0.97.62:38371)
0924 09:56:39.007424 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0924 09:56:39.007430 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 09:56:39.007854 (+ 424us) server_negotiation.cc:407] Connection header received
0924 09:56:39.008002 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:56:39.008007 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:56:39.008092 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:56:39.008237 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:56:39.009628 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:39.010755 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:39.012579 (+ 1824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:39.013143 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:39.013275 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:39.017885 (+ 4610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:56:39.017892 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:56:39.017897 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:56:39.017939 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:56:39.021303 (+ 3364us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:56:39.022086 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:56:39.022092 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:56:39.022096 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:56:39.022253 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:56:39.022652 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:56:39.022656 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:56:39.022659 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:56:39.022991 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:56:39.023226 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0924 09:56:39.023540 (+ 314us) server_negotiation.cc:299] Negotiation successful
0924 09:56:39.023757 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 09:56:39.027871 7998 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36722:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 09:56:39.028522 7998 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 09:56:39.035655 8412 tablet_service.cc:1467] Processing CreateTablet for tablet 25350a1ede214221ab709f3cf2a43ba5 (DEFAULT_TABLE table=test-table [id=c5fac1cad4974dbcb32505a0e039b5a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 09:56:39.035830 8277 tablet_service.cc:1467] Processing CreateTablet for tablet 25350a1ede214221ab709f3cf2a43ba5 (DEFAULT_TABLE table=test-table [id=c5fac1cad4974dbcb32505a0e039b5a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 09:56:39.036131 8412 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25350a1ede214221ab709f3cf2a43ba5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 09:56:39.036260 8277 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25350a1ede214221ab709f3cf2a43ba5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 09:56:39.039585 8134 tablet_service.cc:1467] Processing CreateTablet for tablet 25350a1ede214221ab709f3cf2a43ba5 (DEFAULT_TABLE table=test-table [id=c5fac1cad4974dbcb32505a0e039b5a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 09:56:39.039964 8134 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25350a1ede214221ab709f3cf2a43ba5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 09:56:39.042472 8506 tablet_bootstrap.cc:492] T 25350a1ede214221ab709f3cf2a43ba5 P 818b4870af8045b9ac5ed6fdd83d89dd: Bootstrap starting.
I20260924 09:56:39.044004 8506 tablet_bootstrap.cc:654] T 25350a1ede214221ab709f3cf2a43ba5 P 818b4870af8045b9ac5ed6fdd83d89dd: Neither blocks nor log segments found. Creating new log.
I20260924 09:56:39.048380 8507 tablet_bootstrap.cc:492] T 25350a1ede214221ab709f3cf2a43ba5 P 3e6632b5e12e420283aca499e3c94c42: Bootstrap starting.
I20260924 09:56:39.049844 8507 tablet_bootstrap.cc:654] T 25350a1ede214221ab709f3cf2a43ba5 P 3e6632b5e12e420283aca499e3c94c42: Neither blocks nor log segments found. Creating new log.
I20260924 09:56:39.050763 8506 tablet_bootstrap.cc:492] T 25350a1ede214221ab709f3cf2a43ba5 P 818b4870af8045b9ac5ed6fdd83d89dd: No bootstrap required, opened a new log
I20260924 09:56:39.050917 8506 ts_tablet_manager.cc:1397] T 25350a1ede214221ab709f3cf2a43ba5 P 818b4870af8045b9ac5ed6fdd83d89dd: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.000s
I20260924 09:56:39.051738 8507 tablet_bootstrap.cc:492] T 25350a1ede214221ab709f3cf2a43ba5 P 3e6632b5e12e420283aca499e3c94c42: No bootstrap required, opened a new log
I20260924 09:56:39.051729 8506 raft_consensus.cc:348] T 25350a1ede214221ab709f3cf2a43ba5 P 818b4870af8045b9ac5ed6fdd83d89dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e6632b5e12e420283aca499e3c94c42" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40363 } } peers { permanent_uuid: "43f12639d53d4d19b96ec67379f6fa1f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41195 } } peers { permanent_uuid: "818b4870af8045b9ac5ed6fdd83d89dd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36899 } }
I20260924 09:56:39.052011 8507 ts_tablet_manager.cc:1397] T 25350a1ede214221ab709f3cf2a43ba5 P 3e6632b5e12e420283aca499e3c94c42: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.001s
I20260924 09:56:39.052124 8506 raft_consensus.cc:374] T 25350a1ede214221ab709f3cf2a43ba5 P 818b4870af8045b9ac5ed6fdd83d89dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 09:56:39.052805 8507 raft_consensus.cc:348] T 25350a1ede214221ab709f3cf2a43ba5 P 3e6632b5e12e420283aca499e3c94c42 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e6632b5e12e420283aca499e3c94c42" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40363 } } peers { permanent_uuid: "43f12639d53d4d19b96ec67379f6fa1f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41195 } } peers { permanent_uuid: "818b4870af8045b9ac5ed6fdd83d89dd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36899 } }
I20260924 09:56:39.052269 8506 raft_consensus.cc:729] T 25350a1ede214221ab709f3cf2a43ba5 P 818b4870af8045b9ac5ed6fdd83d89dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 818b4870af8045b9ac5ed6fdd83d89dd, State: Initialized, Role: FOLLOWER
I20260924 09:56:39.053334 8510 tablet_bootstrap.cc:492] T 25350a1ede214221ab709f3cf2a43ba5 P 43f12639d53d4d19b96ec67379f6fa1f: Bootstrap starting.
I20260924 09:56:39.053515 8507 raft_consensus.cc:374] T 25350a1ede214221ab709f3cf2a43ba5 P 3e6632b5e12e420283aca499e3c94c42 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 09:56:39.053612 8507 raft_consensus.cc:729] T 25350a1ede214221ab709f3cf2a43ba5 P 3e6632b5e12e420283aca499e3c94c42 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e6632b5e12e420283aca499e3c94c42, State: Initialized, Role: FOLLOWER
I20260924 09:56:39.053510 8506 consensus_queue.cc:260] T 25350a1ede214221ab709f3cf2a43ba5 P 818b4870af8045b9ac5ed6fdd83d89dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e6632b5e12e420283aca499e3c94c42" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40363 } } peers { permanent_uuid: "43f12639d53d4d19b96ec67379f6fa1f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41195 } } peers { permanent_uuid: "818b4870af8045b9ac5ed6fdd83d89dd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36899 } }
I20260924 09:56:39.053829 8507 consensus_queue.cc:260] T 25350a1ede214221ab709f3cf2a43ba5 P 3e6632b5e12e420283aca499e3c94c42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e6632b5e12e420283aca499e3c94c42" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40363 } } peers { permanent_uuid: "43f12639d53d4d19b96ec67379f6fa1f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41195 } } peers { permanent_uuid: "818b4870af8045b9ac5ed6fdd83d89dd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36899 } }
I20260924 09:56:39.054108 8506 ts_tablet_manager.cc:1428] T 25350a1ede214221ab709f3cf2a43ba5 P 818b4870af8045b9ac5ed6fdd83d89dd: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260924 09:56:39.054934 8510 tablet_bootstrap.cc:654] T 25350a1ede214221ab709f3cf2a43ba5 P 43f12639d53d4d19b96ec67379f6fa1f: Neither blocks nor log segments found. Creating new log.
I20260924 09:56:39.056559 8507 ts_tablet_manager.cc:1428] T 25350a1ede214221ab709f3cf2a43ba5 P 3e6632b5e12e420283aca499e3c94c42: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20260924 09:56:39.060176 8510 tablet_bootstrap.cc:492] T 25350a1ede214221ab709f3cf2a43ba5 P 43f12639d53d4d19b96ec67379f6fa1f: No bootstrap required, opened a new log
I20260924 09:56:39.060324 8510 ts_tablet_manager.cc:1397] T 25350a1ede214221ab709f3cf2a43ba5 P 43f12639d53d4d19b96ec67379f6fa1f: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.004s
I20260924 09:56:39.061061 8510 raft_consensus.cc:348] T 25350a1ede214221ab709f3cf2a43ba5 P 43f12639d53d4d19b96ec67379f6fa1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e6632b5e12e420283aca499e3c94c42" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40363 } } peers { permanent_uuid: "43f12639d53d4d19b96ec67379f6fa1f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41195 } } peers { permanent_uuid: "818b4870af8045b9ac5ed6fdd83d89dd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36899 } }
I20260924 09:56:39.061312 8510 raft_consensus.cc:374] T 25350a1ede214221ab709f3cf2a43ba5 P 43f12639d53d4d19b96ec67379f6fa1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 09:56:39.061409 8510 raft_consensus.cc:729] T 25350a1ede214221ab709f3cf2a43ba5 P 43f12639d53d4d19b96ec67379f6fa1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43f12639d53d4d19b96ec67379f6fa1f, State: Initialized, Role: FOLLOWER
I20260924 09:56:39.061633 8510 consensus_queue.cc:260] T 25350a1ede214221ab709f3cf2a43ba5 P 43f12639d53d4d19b96ec67379f6fa1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e6632b5e12e420283aca499e3c94c42" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40363 } } peers { permanent_uuid: "43f12639d53d4d19b96ec67379f6fa1f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41195 } } peers { permanent_uuid: "818b4870af8045b9ac5ed6fdd83d89dd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36899 } }
I20260924 09:56:39.062309 8510 ts_tablet_manager.cc:1428] T 25350a1ede214221ab709f3cf2a43ba5 P 43f12639d53d4d19b96ec67379f6fa1f: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260924 09:56:39.158125 8514 raft_consensus.cc:482] T 25350a1ede214221ab709f3cf2a43ba5 P 43f12639d53d4d19b96ec67379f6fa1f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 09:56:39.158365 8514 raft_consensus.cc:504] T 25350a1ede214221ab709f3cf2a43ba5 P 43f12639d53d4d19b96ec67379f6fa1f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e6632b5e12e420283aca499e3c94c42" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40363 } } peers { permanent_uuid: "43f12639d53d4d19b96ec67379f6fa1f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41195 } } peers { permanent_uuid: "818b4870af8045b9ac5ed6fdd83d89dd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36899 } }
I20260924 09:56:39.159178 8514 leader_election.cc:288] T 25350a1ede214221ab709f3cf2a43ba5 P 43f12639d53d4d19b96ec67379f6fa1f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3e6632b5e12e420283aca499e3c94c42 (127.0.97.1:40363), 818b4870af8045b9ac5ed6fdd83d89dd (127.0.97.3:36899)
I20260924 09:56:39.159672 8437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25350a1ede214221ab709f3cf2a43ba5" candidate_uuid: "43f12639d53d4d19b96ec67379f6fa1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "818b4870af8045b9ac5ed6fdd83d89dd" is_pre_election: true
I20260924 09:56:39.159675 8153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25350a1ede214221ab709f3cf2a43ba5" candidate_uuid: "43f12639d53d4d19b96ec67379f6fa1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e6632b5e12e420283aca499e3c94c42" is_pre_election: true
I20260924 09:56:39.159914 8153 raft_consensus.cc:2454] T 25350a1ede214221ab709f3cf2a43ba5 P 3e6632b5e12e420283aca499e3c94c42 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 43f12639d53d4d19b96ec67379f6fa1f in term 0.
I20260924 09:56:39.159914 8437 raft_consensus.cc:2454] T 25350a1ede214221ab709f3cf2a43ba5 P 818b4870af8045b9ac5ed6fdd83d89dd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 43f12639d53d4d19b96ec67379f6fa1f in term 0.
I20260924 09:56:39.160348 8225 leader_election.cc:302] T 25350a1ede214221ab709f3cf2a43ba5 P 43f12639d53d4d19b96ec67379f6fa1f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e6632b5e12e420283aca499e3c94c42, 43f12639d53d4d19b96ec67379f6fa1f; no voters:
I20260924 09:56:39.160586 8514 raft_consensus.cc:2781] T 25350a1ede214221ab709f3cf2a43ba5 P 43f12639d53d4d19b96ec67379f6fa1f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 09:56:39.160691 8514 raft_consensus.cc:482] T 25350a1ede214221ab709f3cf2a43ba5 P 43f12639d53d4d19b96ec67379f6fa1f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 09:56:39.160789 8514 raft_consensus.cc:3037] T 25350a1ede214221ab709f3cf2a43ba5 P 43f12639d53d4d19b96ec67379f6fa1f [term 0 FOLLOWER]: Advancing to term 1
I20260924 09:56:39.162886 8514 raft_consensus.cc:504] T 25350a1ede214221ab709f3cf2a43ba5 P 43f12639d53d4d19b96ec67379f6fa1f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e6632b5e12e420283aca499e3c94c42" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40363 } } peers { permanent_uuid: "43f12639d53d4d19b96ec67379f6fa1f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41195 } } peers { permanent_uuid: "818b4870af8045b9ac5ed6fdd83d89dd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36899 } }
I20260924 09:56:39.163414 8514 leader_election.cc:288] T 25350a1ede214221ab709f3cf2a43ba5 P 43f12639d53d4d19b96ec67379f6fa1f [CANDIDATE]: Term 1 election: Requested vote from peers 3e6632b5e12e420283aca499e3c94c42 (127.0.97.1:40363), 818b4870af8045b9ac5ed6fdd83d89dd (127.0.97.3:36899)
I20260924 09:56:39.163789 8153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25350a1ede214221ab709f3cf2a43ba5" candidate_uuid: "43f12639d53d4d19b96ec67379f6fa1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e6632b5e12e420283aca499e3c94c42"
I20260924 09:56:39.163911 8437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25350a1ede214221ab709f3cf2a43ba5" candidate_uuid: "43f12639d53d4d19b96ec67379f6fa1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "818b4870af8045b9ac5ed6fdd83d89dd"
I20260924 09:56:39.163975 8153 raft_consensus.cc:3037] T 25350a1ede214221ab709f3cf2a43ba5 P 3e6632b5e12e420283aca499e3c94c42 [term 0 FOLLOWER]: Advancing to term 1
I20260924 09:56:39.164084 8437 raft_consensus.cc:3037] T 25350a1ede214221ab709f3cf2a43ba5 P 818b4870af8045b9ac5ed6fdd83d89dd [term 0 FOLLOWER]: Advancing to term 1
I20260924 09:56:39.166683 8153 raft_consensus.cc:2454] T 25350a1ede214221ab709f3cf2a43ba5 P 3e6632b5e12e420283aca499e3c94c42 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43f12639d53d4d19b96ec67379f6fa1f in term 1.
I20260924 09:56:39.166683 8437 raft_consensus.cc:2454] T 25350a1ede214221ab709f3cf2a43ba5 P 818b4870af8045b9ac5ed6fdd83d89dd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43f12639d53d4d19b96ec67379f6fa1f in term 1.
I20260924 09:56:39.167021 8225 leader_election.cc:302] T 25350a1ede214221ab709f3cf2a43ba5 P 43f12639d53d4d19b96ec67379f6fa1f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e6632b5e12e420283aca499e3c94c42, 43f12639d53d4d19b96ec67379f6fa1f; no voters:
I20260924 09:56:39.167357 8514 raft_consensus.cc:2781] T 25350a1ede214221ab709f3cf2a43ba5 P 43f12639d53d4d19b96ec67379f6fa1f [term 1 FOLLOWER]: Leader election won for term 1
I20260924 09:56:39.167522 8514 raft_consensus.cc:686] T 25350a1ede214221ab709f3cf2a43ba5 P 43f12639d53d4d19b96ec67379f6fa1f [term 1 LEADER]: Becoming Leader. State: Replica: 43f12639d53d4d19b96ec67379f6fa1f, State: Running, Role: LEADER
I20260924 09:56:39.167781 8514 consensus_queue.cc:237] T 25350a1ede214221ab709f3cf2a43ba5 P 43f12639d53d4d19b96ec67379f6fa1f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e6632b5e12e420283aca499e3c94c42" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40363 } } peers { permanent_uuid: "43f12639d53d4d19b96ec67379f6fa1f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41195 } } peers { permanent_uuid: "818b4870af8045b9ac5ed6fdd83d89dd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36899 } }
I20260924 09:56:39.170390 7998 catalog_manager.cc:5495] T 25350a1ede214221ab709f3cf2a43ba5 P 43f12639d53d4d19b96ec67379f6fa1f reported cstate change: term changed from 0 to 1, leader changed from <none> to 43f12639d53d4d19b96ec67379f6fa1f (127.0.97.2). New cstate: current_term: 1 leader_uuid: "43f12639d53d4d19b96ec67379f6fa1f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e6632b5e12e420283aca499e3c94c42" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40363 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "43f12639d53d4d19b96ec67379f6fa1f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41195 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "818b4870af8045b9ac5ed6fdd83d89dd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36899 } health_report { overall_health: UNKNOWN } } }
I20260924 09:56:39.185134 8505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:39.179814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36852 (local address 127.0.97.2:41195)
0924 09:56:39.179956 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0924 09:56:39.179961 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 09:56:39.180217 (+ 256us) server_negotiation.cc:407] Connection header received
0924 09:56:39.180392 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:56:39.180395 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:56:39.180469 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:56:39.180538 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 09:56:39.180879 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:39.181681 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:39.183586 (+ 1905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:39.184028 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:39.184115 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:39.184531 (+ 416us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 09:56:39.184640 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 09:56:39.184704 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 09:56:39.184830 (+ 126us) server_negotiation.cc:299] Negotiation successful
0924 09:56:39.184889 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 09:56:39.187558 8437 raft_consensus.cc:1261] T 25350a1ede214221ab709f3cf2a43ba5 P 818b4870af8045b9ac5ed6fdd83d89dd [term 1 FOLLOWER]: Refusing update from remote peer 43f12639d53d4d19b96ec67379f6fa1f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 09:56:39.187616 8153 raft_consensus.cc:1261] T 25350a1ede214221ab709f3cf2a43ba5 P 3e6632b5e12e420283aca499e3c94c42 [term 1 FOLLOWER]: Refusing update from remote peer 43f12639d53d4d19b96ec67379f6fa1f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 09:56:39.188107 8554 consensus_queue.cc:1035] T 25350a1ede214221ab709f3cf2a43ba5 P 43f12639d53d4d19b96ec67379f6fa1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "818b4870af8045b9ac5ed6fdd83d89dd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 09:56:39.188361 8514 consensus_queue.cc:1035] T 25350a1ede214221ab709f3cf2a43ba5 P 43f12639d53d4d19b96ec67379f6fa1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e6632b5e12e420283aca499e3c94c42" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40363 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 09:56:39.200294 7998 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36722:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 09:56:39.200553 7998 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36722:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 09:56:39.203351 7998 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e3f28e3acf1e44e59f3ed21ee9c9bd10: Sending DeleteTablet for 3 replicas of tablet 25350a1ede214221ab709f3cf2a43ba5
I20260924 09:56:39.204067 8134 tablet_service.cc:1514] Processing DeleteTablet for tablet 25350a1ede214221ab709f3cf2a43ba5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 09:56:39 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:38830
I20260924 09:56:39.204072 8277 tablet_service.cc:1514] Processing DeleteTablet for tablet 25350a1ede214221ab709f3cf2a43ba5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 09:56:39 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:36820
I20260924 09:56:39.204216 8412 tablet_service.cc:1514] Processing DeleteTablet for tablet 25350a1ede214221ab709f3cf2a43ba5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 09:56:39 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:50590
I20260924 09:56:39.204362 8528 tablet_replica.cc:330] T 25350a1ede214221ab709f3cf2a43ba5 P 3e6632b5e12e420283aca499e3c94c42: stopping tablet replica
I20260924 09:56:39.204501 8530 tablet_replica.cc:330] T 25350a1ede214221ab709f3cf2a43ba5 P 818b4870af8045b9ac5ed6fdd83d89dd: stopping tablet replica
I20260924 09:56:39.204531 8529 tablet_replica.cc:330] T 25350a1ede214221ab709f3cf2a43ba5 P 43f12639d53d4d19b96ec67379f6fa1f: stopping tablet replica
I20260924 09:56:39.204545 8528 raft_consensus.cc:2229] T 25350a1ede214221ab709f3cf2a43ba5 P 3e6632b5e12e420283aca499e3c94c42 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 09:56:39.204656 8530 raft_consensus.cc:2229] T 25350a1ede214221ab709f3cf2a43ba5 P 818b4870af8045b9ac5ed6fdd83d89dd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 09:56:39.204703 8529 raft_consensus.cc:2229] T 25350a1ede214221ab709f3cf2a43ba5 P 43f12639d53d4d19b96ec67379f6fa1f [term 1 LEADER]: Raft consensus shutting down.
I20260924 09:56:39.204770 8528 raft_consensus.cc:2258] T 25350a1ede214221ab709f3cf2a43ba5 P 3e6632b5e12e420283aca499e3c94c42 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 09:56:39.204875 8530 raft_consensus.cc:2258] T 25350a1ede214221ab709f3cf2a43ba5 P 818b4870af8045b9ac5ed6fdd83d89dd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 09:56:39.204994 8529 raft_consensus.cc:2258] T 25350a1ede214221ab709f3cf2a43ba5 P 43f12639d53d4d19b96ec67379f6fa1f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 09:56:39.205816 8530 ts_tablet_manager.cc:1905] T 25350a1ede214221ab709f3cf2a43ba5 P 818b4870af8045b9ac5ed6fdd83d89dd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 09:56:39.206082 8529 ts_tablet_manager.cc:1905] T 25350a1ede214221ab709f3cf2a43ba5 P 43f12639d53d4d19b96ec67379f6fa1f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 09:56:39.210127 8530 ts_tablet_manager.cc:1918] T 25350a1ede214221ab709f3cf2a43ba5 P 818b4870af8045b9ac5ed6fdd83d89dd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 09:56:39.211274 8528 ts_tablet_manager.cc:1905] T 25350a1ede214221ab709f3cf2a43ba5 P 3e6632b5e12e420283aca499e3c94c42: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 09:56:39.212479 8529 ts_tablet_manager.cc:1918] T 25350a1ede214221ab709f3cf2a43ba5 P 43f12639d53d4d19b96ec67379f6fa1f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 09:56:39.212605 8529 log.cc:1196] T 25350a1ede214221ab709f3cf2a43ba5 P 43f12639d53d4d19b96ec67379f6fa1f: Deleting WAL directory at /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790243735634835-388-0/minicluster-data/ts-1/wal/wals/25350a1ede214221ab709f3cf2a43ba5
I20260924 09:56:39.212965 8529 ts_tablet_manager.cc:1939] T 25350a1ede214221ab709f3cf2a43ba5 P 43f12639d53d4d19b96ec67379f6fa1f: Deleting consensus metadata
I20260924 09:56:39.213937 7982 catalog_manager.cc:4841] TS 43f12639d53d4d19b96ec67379f6fa1f (127.0.97.2:41195): tablet 25350a1ede214221ab709f3cf2a43ba5 (table test-table [id=c5fac1cad4974dbcb32505a0e039b5a9]) successfully deleted
I20260924 09:56:39.215194 7998 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36722:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260924 09:56:39.210245 8530 log.cc:1196] T 25350a1ede214221ab709f3cf2a43ba5 P 818b4870af8045b9ac5ed6fdd83d89dd: Deleting WAL directory at /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790243735634835-388-0/minicluster-data/ts-2/wal/wals/25350a1ede214221ab709f3cf2a43ba5
I20260924 09:56:39.215852 8530 ts_tablet_manager.cc:1939] T 25350a1ede214221ab709f3cf2a43ba5 P 818b4870af8045b9ac5ed6fdd83d89dd: Deleting consensus metadata
W20260924 09:56:39.215992 7998 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 09:56:39.216809 7985 catalog_manager.cc:4841] TS 818b4870af8045b9ac5ed6fdd83d89dd (127.0.97.3:36899): tablet 25350a1ede214221ab709f3cf2a43ba5 (table test-table [id=c5fac1cad4974dbcb32505a0e039b5a9]) successfully deleted
I20260924 09:56:39.217928 8528 ts_tablet_manager.cc:1918] T 25350a1ede214221ab709f3cf2a43ba5 P 3e6632b5e12e420283aca499e3c94c42: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 09:56:39.218077 8528 log.cc:1196] T 25350a1ede214221ab709f3cf2a43ba5 P 3e6632b5e12e420283aca499e3c94c42: Deleting WAL directory at /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790243735634835-388-0/minicluster-data/ts-0/wal/wals/25350a1ede214221ab709f3cf2a43ba5
I20260924 09:56:39.218442 8528 ts_tablet_manager.cc:1939] T 25350a1ede214221ab709f3cf2a43ba5 P 3e6632b5e12e420283aca499e3c94c42: Deleting consensus metadata
I20260924 09:56:39.219352 7983 catalog_manager.cc:4841] TS 3e6632b5e12e420283aca499e3c94c42 (127.0.97.1:40363): tablet 25350a1ede214221ab709f3cf2a43ba5 (table test-table [id=c5fac1cad4974dbcb32505a0e039b5a9]) successfully deleted
I20260924 09:56:39.222599 8134 tablet_service.cc:1467] Processing CreateTablet for tablet cc461f91e5c54e358c649fb3e5680c46 (DEFAULT_TABLE table=test-table [id=19285d0b43534ba88a353a379bd407f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 09:56:39.222756 8277 tablet_service.cc:1467] Processing CreateTablet for tablet cc461f91e5c54e358c649fb3e5680c46 (DEFAULT_TABLE table=test-table [id=19285d0b43534ba88a353a379bd407f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 09:56:39.222968 8134 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc461f91e5c54e358c649fb3e5680c46. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 09:56:39.223173 8277 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc461f91e5c54e358c649fb3e5680c46. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 09:56:39.227139 8507 tablet_bootstrap.cc:492] T cc461f91e5c54e358c649fb3e5680c46 P 3e6632b5e12e420283aca499e3c94c42: Bootstrap starting.
I20260924 09:56:39.227612 8412 tablet_service.cc:1467] Processing CreateTablet for tablet cc461f91e5c54e358c649fb3e5680c46 (DEFAULT_TABLE table=test-table [id=19285d0b43534ba88a353a379bd407f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 09:56:39.227993 8412 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc461f91e5c54e358c649fb3e5680c46. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 09:56:39.228525 8507 tablet_bootstrap.cc:654] T cc461f91e5c54e358c649fb3e5680c46 P 3e6632b5e12e420283aca499e3c94c42: Neither blocks nor log segments found. Creating new log.
I20260924 09:56:39.229979 8507 tablet_bootstrap.cc:492] T cc461f91e5c54e358c649fb3e5680c46 P 3e6632b5e12e420283aca499e3c94c42: No bootstrap required, opened a new log
I20260924 09:56:39.230114 8507 ts_tablet_manager.cc:1397] T cc461f91e5c54e358c649fb3e5680c46 P 3e6632b5e12e420283aca499e3c94c42: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.001s
I20260924 09:56:39.230652 8507 raft_consensus.cc:348] T cc461f91e5c54e358c649fb3e5680c46 P 3e6632b5e12e420283aca499e3c94c42 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e6632b5e12e420283aca499e3c94c42" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40363 } } peers { permanent_uuid: "43f12639d53d4d19b96ec67379f6fa1f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41195 } } peers { permanent_uuid: "818b4870af8045b9ac5ed6fdd83d89dd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36899 } }
I20260924 09:56:39.230834 8507 raft_consensus.cc:374] T cc461f91e5c54e358c649fb3e5680c46 P 3e6632b5e12e420283aca499e3c94c42 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 09:56:39.230904 8507 raft_consensus.cc:729] T cc461f91e5c54e358c649fb3e5680c46 P 3e6632b5e12e420283aca499e3c94c42 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e6632b5e12e420283aca499e3c94c42, State: Initialized, Role: FOLLOWER
I20260924 09:56:39.231109 8507 consensus_queue.cc:260] T cc461f91e5c54e358c649fb3e5680c46 P 3e6632b5e12e420283aca499e3c94c42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e6632b5e12e420283aca499e3c94c42" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40363 } } peers { permanent_uuid: "43f12639d53d4d19b96ec67379f6fa1f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41195 } } peers { permanent_uuid: "818b4870af8045b9ac5ed6fdd83d89dd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36899 } }
I20260924 09:56:39.231578 8507 ts_tablet_manager.cc:1428] T cc461f91e5c54e358c649fb3e5680c46 P 3e6632b5e12e420283aca499e3c94c42: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260924 09:56:39.234730 8510 tablet_bootstrap.cc:492] T cc461f91e5c54e358c649fb3e5680c46 P 43f12639d53d4d19b96ec67379f6fa1f: Bootstrap starting.
I20260924 09:56:39.236390 8510 tablet_bootstrap.cc:654] T cc461f91e5c54e358c649fb3e5680c46 P 43f12639d53d4d19b96ec67379f6fa1f: Neither blocks nor log segments found. Creating new log.
I20260924 09:56:39.236974 8506 tablet_bootstrap.cc:492] T cc461f91e5c54e358c649fb3e5680c46 P 818b4870af8045b9ac5ed6fdd83d89dd: Bootstrap starting.
I20260924 09:56:39.238565 8506 tablet_bootstrap.cc:654] T cc461f91e5c54e358c649fb3e5680c46 P 818b4870af8045b9ac5ed6fdd83d89dd: Neither blocks nor log segments found. Creating new log.
I20260924 09:56:39.241504 8506 tablet_bootstrap.cc:492] T cc461f91e5c54e358c649fb3e5680c46 P 818b4870af8045b9ac5ed6fdd83d89dd: No bootstrap required, opened a new log
I20260924 09:56:39.241660 8506 ts_tablet_manager.cc:1397] T cc461f91e5c54e358c649fb3e5680c46 P 818b4870af8045b9ac5ed6fdd83d89dd: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260924 09:56:39.241747 8510 tablet_bootstrap.cc:492] T cc461f91e5c54e358c649fb3e5680c46 P 43f12639d53d4d19b96ec67379f6fa1f: No bootstrap required, opened a new log
I20260924 09:56:39.241899 8510 ts_tablet_manager.cc:1397] T cc461f91e5c54e358c649fb3e5680c46 P 43f12639d53d4d19b96ec67379f6fa1f: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.004s
I20260924 09:56:39.242359 8506 raft_consensus.cc:348] T cc461f91e5c54e358c649fb3e5680c46 P 818b4870af8045b9ac5ed6fdd83d89dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e6632b5e12e420283aca499e3c94c42" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40363 } } peers { permanent_uuid: "43f12639d53d4d19b96ec67379f6fa1f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41195 } } peers { permanent_uuid: "818b4870af8045b9ac5ed6fdd83d89dd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36899 } }
I20260924 09:56:39.242592 8506 raft_consensus.cc:374] T cc461f91e5c54e358c649fb3e5680c46 P 818b4870af8045b9ac5ed6fdd83d89dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 09:56:39.242667 8506 raft_consensus.cc:729] T cc461f91e5c54e358c649fb3e5680c46 P 818b4870af8045b9ac5ed6fdd83d89dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 818b4870af8045b9ac5ed6fdd83d89dd, State: Initialized, Role: FOLLOWER
I20260924 09:56:39.242584 8510 raft_consensus.cc:348] T cc461f91e5c54e358c649fb3e5680c46 P 43f12639d53d4d19b96ec67379f6fa1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e6632b5e12e420283aca499e3c94c42" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40363 } } peers { permanent_uuid: "43f12639d53d4d19b96ec67379f6fa1f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41195 } } peers { permanent_uuid: "818b4870af8045b9ac5ed6fdd83d89dd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36899 } }
I20260924 09:56:39.242892 8510 raft_consensus.cc:374] T cc461f91e5c54e358c649fb3e5680c46 P 43f12639d53d4d19b96ec67379f6fa1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 09:56:39.242892 8506 consensus_queue.cc:260] T cc461f91e5c54e358c649fb3e5680c46 P 818b4870af8045b9ac5ed6fdd83d89dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e6632b5e12e420283aca499e3c94c42" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40363 } } peers { permanent_uuid: "43f12639d53d4d19b96ec67379f6fa1f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41195 } } peers { permanent_uuid: "818b4870af8045b9ac5ed6fdd83d89dd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36899 } }
I20260924 09:56:39.243016 8510 raft_consensus.cc:729] T cc461f91e5c54e358c649fb3e5680c46 P 43f12639d53d4d19b96ec67379f6fa1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43f12639d53d4d19b96ec67379f6fa1f, State: Initialized, Role: FOLLOWER
I20260924 09:56:39.243400 8506 ts_tablet_manager.cc:1428] T cc461f91e5c54e358c649fb3e5680c46 P 818b4870af8045b9ac5ed6fdd83d89dd: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 09:56:39.243299 8510 consensus_queue.cc:260] T cc461f91e5c54e358c649fb3e5680c46 P 43f12639d53d4d19b96ec67379f6fa1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e6632b5e12e420283aca499e3c94c42" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40363 } } peers { permanent_uuid: "43f12639d53d4d19b96ec67379f6fa1f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41195 } } peers { permanent_uuid: "818b4870af8045b9ac5ed6fdd83d89dd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36899 } }
I20260924 09:56:39.243745 8510 ts_tablet_manager.cc:1428] T cc461f91e5c54e358c649fb3e5680c46 P 43f12639d53d4d19b96ec67379f6fa1f: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260924 09:56:39.306171 8512 raft_consensus.cc:482] T cc461f91e5c54e358c649fb3e5680c46 P 818b4870af8045b9ac5ed6fdd83d89dd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 09:56:39.306344 8512 raft_consensus.cc:504] T cc461f91e5c54e358c649fb3e5680c46 P 818b4870af8045b9ac5ed6fdd83d89dd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e6632b5e12e420283aca499e3c94c42" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40363 } } peers { permanent_uuid: "43f12639d53d4d19b96ec67379f6fa1f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41195 } } peers { permanent_uuid: "818b4870af8045b9ac5ed6fdd83d89dd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36899 } }
I20260924 09:56:39.306963 8512 leader_election.cc:288] T cc461f91e5c54e358c649fb3e5680c46 P 818b4870af8045b9ac5ed6fdd83d89dd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3e6632b5e12e420283aca499e3c94c42 (127.0.97.1:40363), 43f12639d53d4d19b96ec67379f6fa1f (127.0.97.2:41195)
I20260924 09:56:39.307477 8153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc461f91e5c54e358c649fb3e5680c46" candidate_uuid: "818b4870af8045b9ac5ed6fdd83d89dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e6632b5e12e420283aca499e3c94c42" is_pre_election: true
I20260924 09:56:39.307526 8298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc461f91e5c54e358c649fb3e5680c46" candidate_uuid: "818b4870af8045b9ac5ed6fdd83d89dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43f12639d53d4d19b96ec67379f6fa1f" is_pre_election: true
I20260924 09:56:39.307739 8153 raft_consensus.cc:2454] T cc461f91e5c54e358c649fb3e5680c46 P 3e6632b5e12e420283aca499e3c94c42 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818b4870af8045b9ac5ed6fdd83d89dd in term 0.
I20260924 09:56:39.307770 8298 raft_consensus.cc:2454] T cc461f91e5c54e358c649fb3e5680c46 P 43f12639d53d4d19b96ec67379f6fa1f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818b4870af8045b9ac5ed6fdd83d89dd in term 0.
I20260924 09:56:39.308158 8363 leader_election.cc:302] T cc461f91e5c54e358c649fb3e5680c46 P 818b4870af8045b9ac5ed6fdd83d89dd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43f12639d53d4d19b96ec67379f6fa1f, 818b4870af8045b9ac5ed6fdd83d89dd; no voters:
I20260924 09:56:39.308439 8512 raft_consensus.cc:2781] T cc461f91e5c54e358c649fb3e5680c46 P 818b4870af8045b9ac5ed6fdd83d89dd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 09:56:39.308557 8512 raft_consensus.cc:482] T cc461f91e5c54e358c649fb3e5680c46 P 818b4870af8045b9ac5ed6fdd83d89dd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 09:56:39.308650 8512 raft_consensus.cc:3037] T cc461f91e5c54e358c649fb3e5680c46 P 818b4870af8045b9ac5ed6fdd83d89dd [term 0 FOLLOWER]: Advancing to term 1
I20260924 09:56:39.311573 8512 raft_consensus.cc:504] T cc461f91e5c54e358c649fb3e5680c46 P 818b4870af8045b9ac5ed6fdd83d89dd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e6632b5e12e420283aca499e3c94c42" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40363 } } peers { permanent_uuid: "43f12639d53d4d19b96ec67379f6fa1f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41195 } } peers { permanent_uuid: "818b4870af8045b9ac5ed6fdd83d89dd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36899 } }
I20260924 09:56:39.312222 8512 leader_election.cc:288] T cc461f91e5c54e358c649fb3e5680c46 P 818b4870af8045b9ac5ed6fdd83d89dd [CANDIDATE]: Term 1 election: Requested vote from peers 3e6632b5e12e420283aca499e3c94c42 (127.0.97.1:40363), 43f12639d53d4d19b96ec67379f6fa1f (127.0.97.2:41195)
I20260924 09:56:39.312639 8153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc461f91e5c54e358c649fb3e5680c46" candidate_uuid: "818b4870af8045b9ac5ed6fdd83d89dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e6632b5e12e420283aca499e3c94c42"
I20260924 09:56:39.312849 8153 raft_consensus.cc:3037] T cc461f91e5c54e358c649fb3e5680c46 P 3e6632b5e12e420283aca499e3c94c42 [term 0 FOLLOWER]: Advancing to term 1
I20260924 09:56:39.312938 8298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc461f91e5c54e358c649fb3e5680c46" candidate_uuid: "818b4870af8045b9ac5ed6fdd83d89dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43f12639d53d4d19b96ec67379f6fa1f"
I20260924 09:56:39.313122 8298 raft_consensus.cc:3037] T cc461f91e5c54e358c649fb3e5680c46 P 43f12639d53d4d19b96ec67379f6fa1f [term 0 FOLLOWER]: Advancing to term 1
I20260924 09:56:39.315088 8298 raft_consensus.cc:2454] T cc461f91e5c54e358c649fb3e5680c46 P 43f12639d53d4d19b96ec67379f6fa1f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 818b4870af8045b9ac5ed6fdd83d89dd in term 1.
I20260924 09:56:39.315454 8363 leader_election.cc:302] T cc461f91e5c54e358c649fb3e5680c46 P 818b4870af8045b9ac5ed6fdd83d89dd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43f12639d53d4d19b96ec67379f6fa1f, 818b4870af8045b9ac5ed6fdd83d89dd; no voters:
I20260924 09:56:39.315596 8153 raft_consensus.cc:2454] T cc461f91e5c54e358c649fb3e5680c46 P 3e6632b5e12e420283aca499e3c94c42 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 818b4870af8045b9ac5ed6fdd83d89dd in term 1.
I20260924 09:56:39.315721 8512 raft_consensus.cc:2781] T cc461f91e5c54e358c649fb3e5680c46 P 818b4870af8045b9ac5ed6fdd83d89dd [term 1 FOLLOWER]: Leader election won for term 1
I20260924 09:56:39.316258 8512 raft_consensus.cc:686] T cc461f91e5c54e358c649fb3e5680c46 P 818b4870af8045b9ac5ed6fdd83d89dd [term 1 LEADER]: Becoming Leader. State: Replica: 818b4870af8045b9ac5ed6fdd83d89dd, State: Running, Role: LEADER
I20260924 09:56:39.316697 8512 consensus_queue.cc:237] T cc461f91e5c54e358c649fb3e5680c46 P 818b4870af8045b9ac5ed6fdd83d89dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e6632b5e12e420283aca499e3c94c42" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40363 } } peers { permanent_uuid: "43f12639d53d4d19b96ec67379f6fa1f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41195 } } peers { permanent_uuid: "818b4870af8045b9ac5ed6fdd83d89dd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36899 } }
I20260924 09:56:39.319094 7997 catalog_manager.cc:5495] T cc461f91e5c54e358c649fb3e5680c46 P 818b4870af8045b9ac5ed6fdd83d89dd reported cstate change: term changed from 0 to 1, leader changed from <none> to 818b4870af8045b9ac5ed6fdd83d89dd (127.0.97.3). New cstate: current_term: 1 leader_uuid: "818b4870af8045b9ac5ed6fdd83d89dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e6632b5e12e420283aca499e3c94c42" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40363 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "43f12639d53d4d19b96ec67379f6fa1f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "818b4870af8045b9ac5ed6fdd83d89dd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36899 } health_report { overall_health: HEALTHY } } }
I20260924 09:56:39.349659 8586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:39.344366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50606 (local address 127.0.97.3:36899)
0924 09:56:39.344912 (+ 546us) server_negotiation.cc:206] Beginning negotiation
0924 09:56:39.344917 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 09:56:39.344936 (+ 19us) server_negotiation.cc:407] Connection header received
0924 09:56:39.344994 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:56:39.344999 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:56:39.345074 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:56:39.345173 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 09:56:39.345463 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:39.346582 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:39.347878 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:39.348515 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:39.348639 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:39.348947 (+ 308us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 09:56:39.349066 (+ 119us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 09:56:39.349153 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0924 09:56:39.349240 (+ 87us) server_negotiation.cc:299] Negotiation successful
0924 09:56:39.349314 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":430,"thread_start_us":302,"threads_started":1}
I20260924 09:56:39.364488 8505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:39.358755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:37589 (local address 127.0.97.2:41195)
0924 09:56:39.358941 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0924 09:56:39.358948 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 09:56:39.359333 (+ 385us) server_negotiation.cc:407] Connection header received
0924 09:56:39.359530 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:56:39.359535 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:56:39.359615 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:56:39.359685 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 09:56:39.360164 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:39.360998 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:39.362889 (+ 1891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:39.363383 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:39.363466 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:39.363933 (+ 467us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 09:56:39.364025 (+ 92us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 09:56:39.364083 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 09:56:39.364203 (+ 120us) server_negotiation.cc:299] Negotiation successful
0924 09:56:39.364257 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 09:56:39.364686 8589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:39.358663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:41195 (local address 127.0.97.3:37589)
0924 09:56:39.359228 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0924 09:56:39.359270 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 09:56:39.359399 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:56:39.359716 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:56:39.359720 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:56:39.359733 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 09:56:39.360031 (+ 298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:39.360041 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:39.361121 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:39.361127 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:39.362772 (+ 1645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:39.362785 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:39.363532 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:39.363538 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:39.363717 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:39.363781 (+ 64us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 09:56:39.364107 (+ 326us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 09:56:39.364117 (+ 10us) client_negotiation.cc:769] Sending connection context
0924 09:56:39.364193 (+ 76us) client_negotiation.cc:240] Negotiation successful
0924 09:56:39.364321 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":287,"threads_started":1}
I20260924 09:56:39.372373 8298 raft_consensus.cc:1261] T cc461f91e5c54e358c649fb3e5680c46 P 43f12639d53d4d19b96ec67379f6fa1f [term 1 FOLLOWER]: Refusing update from remote peer 818b4870af8045b9ac5ed6fdd83d89dd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 09:56:39.372560 8153 raft_consensus.cc:1261] T cc461f91e5c54e358c649fb3e5680c46 P 3e6632b5e12e420283aca499e3c94c42 [term 1 FOLLOWER]: Refusing update from remote peer 818b4870af8045b9ac5ed6fdd83d89dd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 09:56:39.372972 8512 consensus_queue.cc:1035] T cc461f91e5c54e358c649fb3e5680c46 P 818b4870af8045b9ac5ed6fdd83d89dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "43f12639d53d4d19b96ec67379f6fa1f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 09:56:39.373273 8583 consensus_queue.cc:1035] T cc461f91e5c54e358c649fb3e5680c46 P 818b4870af8045b9ac5ed6fdd83d89dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e6632b5e12e420283aca499e3c94c42" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40363 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 09:56:39.410305 8586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:39.404194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:46065 (local address 127.0.97.3:36899)
0924 09:56:39.404584 (+ 390us) server_negotiation.cc:206] Beginning negotiation
0924 09:56:39.404590 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 09:56:39.405122 (+ 532us) server_negotiation.cc:407] Connection header received
0924 09:56:39.405308 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:56:39.405313 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:56:39.405373 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:56:39.405440 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 09:56:39.405880 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:39.406555 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:39.408523 (+ 1968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:39.409030 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:39.409140 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:39.409664 (+ 524us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 09:56:39.409788 (+ 124us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 09:56:39.409868 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0924 09:56:39.409972 (+ 104us) server_negotiation.cc:299] Negotiation successful
0924 09:56:39.410049 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":253}
I20260924 09:56:39.410332 8594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:39.404109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:36899 (local address 127.0.97.2:46065)
0924 09:56:39.405036 (+ 927us) negotiation.cc:107] Waiting for socket to connect
0924 09:56:39.405054 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 09:56:39.405176 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:56:39.405483 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:56:39.405488 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:56:39.405502 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 09:56:39.405734 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:39.405744 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:39.406666 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:39.406672 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:39.408385 (+ 1713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:39.408403 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:39.409256 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:39.409266 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:39.409464 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:39.409528 (+ 64us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 09:56:39.409887 (+ 359us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 09:56:39.409896 (+ 9us) client_negotiation.cc:769] Sending connection context
0924 09:56:39.409949 (+ 53us) client_negotiation.cc:240] Negotiation successful
0924 09:56:39.410046 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":790,"thread_start_us":648,"threads_started":1}
I20260924 09:56:39.448483 7998 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36722:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 09:56:39.448684 7998 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36722:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 09:56:39.451395 7998 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e3f28e3acf1e44e59f3ed21ee9c9bd10: Sending DeleteTablet for 3 replicas of tablet cc461f91e5c54e358c649fb3e5680c46
I20260924 09:56:39.452101 8134 tablet_service.cc:1514] Processing DeleteTablet for tablet cc461f91e5c54e358c649fb3e5680c46 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 09:56:39 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:38830
I20260924 09:56:39.452385 8528 tablet_replica.cc:330] T cc461f91e5c54e358c649fb3e5680c46 P 3e6632b5e12e420283aca499e3c94c42: stopping tablet replica
I20260924 09:56:39.452338 8412 tablet_service.cc:1514] Processing DeleteTablet for tablet cc461f91e5c54e358c649fb3e5680c46 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 09:56:39 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:50590
I20260924 09:56:39.452571 8530 tablet_replica.cc:330] T cc461f91e5c54e358c649fb3e5680c46 P 818b4870af8045b9ac5ed6fdd83d89dd: stopping tablet replica
I20260924 09:56:39.452565 8528 raft_consensus.cc:2229] T cc461f91e5c54e358c649fb3e5680c46 P 3e6632b5e12e420283aca499e3c94c42 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 09:56:39.452732 8528 raft_consensus.cc:2258] T cc461f91e5c54e358c649fb3e5680c46 P 3e6632b5e12e420283aca499e3c94c42 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 09:56:39.452749 8530 raft_consensus.cc:2229] T cc461f91e5c54e358c649fb3e5680c46 P 818b4870af8045b9ac5ed6fdd83d89dd [term 1 LEADER]: Raft consensus shutting down.
I20260924 09:56:39.452798 8277 tablet_service.cc:1514] Processing DeleteTablet for tablet cc461f91e5c54e358c649fb3e5680c46 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 09:56:39 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:36820
I20260924 09:56:39.453047 8529 tablet_replica.cc:330] T cc461f91e5c54e358c649fb3e5680c46 P 43f12639d53d4d19b96ec67379f6fa1f: stopping tablet replica
I20260924 09:56:39.453043 8530 raft_consensus.cc:2258] T cc461f91e5c54e358c649fb3e5680c46 P 818b4870af8045b9ac5ed6fdd83d89dd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 09:56:39.453222 8529 raft_consensus.cc:2229] T cc461f91e5c54e358c649fb3e5680c46 P 43f12639d53d4d19b96ec67379f6fa1f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 09:56:39.453392 8529 raft_consensus.cc:2258] T cc461f91e5c54e358c649fb3e5680c46 P 43f12639d53d4d19b96ec67379f6fa1f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 09:56:39.453999 8528 ts_tablet_manager.cc:1905] T cc461f91e5c54e358c649fb3e5680c46 P 3e6632b5e12e420283aca499e3c94c42: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 09:56:39.454257 8530 ts_tablet_manager.cc:1905] T cc461f91e5c54e358c649fb3e5680c46 P 818b4870af8045b9ac5ed6fdd83d89dd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 09:56:39.455149 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu pid 8069
I20260924 09:56:39.457726 8529 ts_tablet_manager.cc:1905] T cc461f91e5c54e358c649fb3e5680c46 P 43f12639d53d4d19b96ec67379f6fa1f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 09:56:39.460832 8528 ts_tablet_manager.cc:1918] T cc461f91e5c54e358c649fb3e5680c46 P 3e6632b5e12e420283aca499e3c94c42: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 09:56:39.461032 8528 log.cc:1196] T cc461f91e5c54e358c649fb3e5680c46 P 3e6632b5e12e420283aca499e3c94c42: Deleting WAL directory at /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790243735634835-388-0/minicluster-data/ts-0/wal/wals/cc461f91e5c54e358c649fb3e5680c46
I20260924 09:56:39.461504 8528 ts_tablet_manager.cc:1939] T cc461f91e5c54e358c649fb3e5680c46 P 3e6632b5e12e420283aca499e3c94c42: Deleting consensus metadata
I20260924 09:56:39.462482 7983 catalog_manager.cc:4841] TS 3e6632b5e12e420283aca499e3c94c42 (127.0.97.1:40363): tablet cc461f91e5c54e358c649fb3e5680c46 (table test-table [id=19285d0b43534ba88a353a379bd407f0]) successfully deleted
I20260924 09:56:39.463757 8529 ts_tablet_manager.cc:1918] T cc461f91e5c54e358c649fb3e5680c46 P 43f12639d53d4d19b96ec67379f6fa1f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 09:56:39.463889 8529 log.cc:1196] T cc461f91e5c54e358c649fb3e5680c46 P 43f12639d53d4d19b96ec67379f6fa1f: Deleting WAL directory at /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790243735634835-388-0/minicluster-data/ts-1/wal/wals/cc461f91e5c54e358c649fb3e5680c46
I20260924 09:56:39.464227 8529 ts_tablet_manager.cc:1939] T cc461f91e5c54e358c649fb3e5680c46 P 43f12639d53d4d19b96ec67379f6fa1f: Deleting consensus metadata
I20260924 09:56:39.464679 8530 ts_tablet_manager.cc:1918] T cc461f91e5c54e358c649fb3e5680c46 P 818b4870af8045b9ac5ed6fdd83d89dd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 09:56:39.464815 8530 log.cc:1196] T cc461f91e5c54e358c649fb3e5680c46 P 818b4870af8045b9ac5ed6fdd83d89dd: Deleting WAL directory at /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790243735634835-388-0/minicluster-data/ts-2/wal/wals/cc461f91e5c54e358c649fb3e5680c46
I20260924 09:56:39.464941 7982 catalog_manager.cc:4841] TS 43f12639d53d4d19b96ec67379f6fa1f (127.0.97.2:41195): tablet cc461f91e5c54e358c649fb3e5680c46 (table test-table [id=19285d0b43534ba88a353a379bd407f0]) successfully deleted
I20260924 09:56:39.465157 8530 ts_tablet_manager.cc:1939] T cc461f91e5c54e358c649fb3e5680c46 P 818b4870af8045b9ac5ed6fdd83d89dd: Deleting consensus metadata
I20260924 09:56:39.466022 7985 catalog_manager.cc:4841] TS 818b4870af8045b9ac5ed6fdd83d89dd (127.0.97.3:36899): tablet cc461f91e5c54e358c649fb3e5680c46 (table test-table [id=19285d0b43534ba88a353a379bd407f0]) successfully deleted
Sep 24 09:56:39 dist-test-slave-w8d2 krb5kdc[7953](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243798, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.1@KRBTEST.COM
I20260924 09:56:39.482373 8598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:39.456335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38832 (local address 127.0.97.1:40363)
0924 09:56:39.467567 (+ 11232us) server_negotiation.cc:206] Beginning negotiation
0924 09:56:39.467571 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 09:56:39.467599 (+ 28us) server_negotiation.cc:407] Connection header received
0924 09:56:39.467642 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:56:39.467645 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:56:39.467699 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:56:39.467781 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:56:39.469315 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:39.470091 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:39.471792 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:39.472165 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:39.472249 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:39.476769 (+ 4520us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:56:39.476787 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:56:39.476807 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:56:39.476842 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:56:39.479637 (+ 2795us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:56:39.480417 (+ 780us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:56:39.480424 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:56:39.480426 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:56:39.480563 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:56:39.481016 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:56:39.481019 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:56:39.481021 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:56:39.481360 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:56:39.481593 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0924 09:56:39.481909 (+ 316us) server_negotiation.cc:299] Negotiation successful
0924 09:56:39.482093 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11122,"thread_start_us":11040,"threads_started":1}
I20260924 09:56:39.482975 8199 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 09:56:39.577576 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu with pid 8069
I20260924 09:56:39.599009 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu pid 8208
Sep 24 09:56:39 dist-test-slave-w8d2 krb5kdc[7953](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243798, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.2@KRBTEST.COM
I20260924 09:56:39.618784 8505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:39.599959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36866 (local address 127.0.97.2:41195)
0924 09:56:39.600124 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0924 09:56:39.600130 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 09:56:39.600150 (+ 20us) server_negotiation.cc:407] Connection header received
0924 09:56:39.600287 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:56:39.600292 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:56:39.600366 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:56:39.600473 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:56:39.601949 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:39.603062 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:39.604903 (+ 1841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:39.605660 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:39.605781 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:39.610862 (+ 5081us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:56:39.610880 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:56:39.610900 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:56:39.610939 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:56:39.614830 (+ 3891us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:56:39.615876 (+ 1046us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:56:39.615885 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:56:39.615889 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:56:39.616051 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:56:39.616451 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:56:39.616456 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:56:39.616459 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:56:39.616877 (+ 418us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:56:39.617167 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0924 09:56:39.618194 (+ 1027us) server_negotiation.cc:299] Negotiation successful
0924 09:56:39.618410 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 09:56:39.619483 8338 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 09:56:39.773842 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu with pid 8208
I20260924 09:56:39.793495 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu pid 8347
Sep 24 09:56:39 dist-test-slave-w8d2 krb5kdc[7953](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243798, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.3@KRBTEST.COM
I20260924 09:56:39.807435 8586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:39.794243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50616 (local address 127.0.97.3:36899)
0924 09:56:39.794440 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0924 09:56:39.794446 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 09:56:39.794497 (+ 51us) server_negotiation.cc:407] Connection header received
0924 09:56:39.794695 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:56:39.794699 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:56:39.794779 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:56:39.794878 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:56:39.796138 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:39.796836 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:39.798058 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:39.798474 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:39.798582 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:39.802431 (+ 3849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:56:39.802447 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:56:39.802462 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:56:39.802497 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:56:39.804975 (+ 2478us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:56:39.805580 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:56:39.805587 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:56:39.805590 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:56:39.805714 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:56:39.806034 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:56:39.806039 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:56:39.806041 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:56:39.806353 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:56:39.806633 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0924 09:56:39.806890 (+ 257us) server_negotiation.cc:299] Negotiation successful
0924 09:56:39.807084 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260924 09:56:39.808061 8477 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 09:56:39.928006 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu with pid 8347
I20260924 09:56:39.945045 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu pid 7966
I20260924 09:56:39.946199 8038 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 09:56:40.056473 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu with pid 7966
2026-09-24T09:56:40Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (5034 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790243735634835-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 09:56:40 dist-test-slave-w8d2 krb5kdc[8605](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 09:56:40 dist-test-slave-w8d2 krb5kdc[8605](info): set up 2 sockets
Sep 24 09:56:40 dist-test-slave-w8d2 krb5kdc[8605](info): commencing operation
krb5kdc: starting...
W20260924 09:56:41.051937 388 mini_kdc.cc:121] Time spent starting KDC: real 0.956s user 0.000s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790243735634835-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790243735634835-388-0/krb5kdc/test-user.keytab.
Sep 24 09:56:41 dist-test-slave-w8d2 krb5kdc[8605](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243801, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T09:56:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T09:56:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790243735634835-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790243735634835-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260924 09:56:41.243151 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
/tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790243735634835-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790243735634835-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790243735634835-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790243735634835-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33093
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790243735634835-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:33093
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790243735634835-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790243735634835-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790243735634835-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 09:56:41.544989 8618 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 09:56:41.545279 8618 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 09:56:41.545415 8618 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 09:56:41.552470 8618 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 09:56:41.552583 8618 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 09:56:41.552659 8618 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 09:56:41.552732 8618 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 09:56:41.552794 8618 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 09:56:41.562299 8618 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790243735634835-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790243735634835-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:33093
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790243735634835-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33093
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790243735634835-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790243735634835-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 09:56:41.563539 8618 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 09:56:41.564756 8618 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 09:56:41.575660 8623 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:56:41.575620 8624 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:56:41.575632 8626 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 09:56:41.576391 8618 server_base.cc:1034] running on GCE node
I20260924 09:56:41.577072 8618 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 09:56:41.578351 8618 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 09:56:41.579541 8618 hybrid_clock.cc:648] HybridClock initialized: now 1790243801579493 us; error 39 us; skew 500 ppm
I20260924 09:56:41.579861 8618 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-24T09:56:41Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1592 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790243735634835-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 09:56:41 dist-test-slave-w8d2 krb5kdc[8632](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 09:56:41 dist-test-slave-w8d2 krb5kdc[8632](info): set up 2 sockets
Sep 24 09:56:41 dist-test-slave-w8d2 krb5kdc[8632](info): commencing operation
krb5kdc: starting...
W20260924 09:56:42.643915 388 mini_kdc.cc:121] Time spent starting KDC: real 0.956s user 0.001s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790243735634835-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790243735634835-388-0/krb5kdc/test-user.keytab.
Sep 24 09:56:42 dist-test-slave-w8d2 krb5kdc[8632](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243802, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T09:56:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T09:56:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790243735634835-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790243735634835-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260924 09:56:42.840724 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
/tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790243735634835-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790243735634835-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790243735634835-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790243735634835-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34237
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790243735634835-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:34237
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790243735634835-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790243735634835-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790243735634835-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 09:56:43.149678 8645 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 09:56:43.149942 8645 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 09:56:43.150071 8645 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 09:56:43.157015 8645 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 09:56:43.157166 8645 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 09:56:43.157238 8645 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 09:56:43.157310 8645 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 09:56:43.157364 8645 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 09:56:43.166800 8645 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790243735634835-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790243735634835-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:34237
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790243735634835-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34237
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790243735634835-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790243735634835-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 09:56:43.168046 8645 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 09:56:43.169355 8645 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 09:56:43.179981 8650 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:56:43.180009 8651 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:56:43.180186 8653 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 09:56:43.180624 8645 server_base.cc:1034] running on GCE node
I20260924 09:56:43.181272 8645 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 09:56:43.182515 8645 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 09:56:43.183710 8645 hybrid_clock.cc:648] HybridClock initialized: now 1790243803183646 us; error 46 us; skew 500 ppm
I20260924 09:56:43.184021 8645 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 09:56:43 dist-test-slave-w8d2 krb5kdc[8632](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243803, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 09:56:43.186764 8645 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260924 09:56:43.188187 8645 webserver.cc:460] Webserver started at http://127.0.97.62:41573/ using document root <none> and password file <none>
I20260924 09:56:43.188807 8645 fs_manager.cc:359] Metadata directory not provided
I20260924 09:56:43.188912 8645 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 09:56:43.189177 8645 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 09:56:43.191946 8645 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790243735634835-388-0/minicluster-data/master-0/data/instance:
uuid: "4ef3a6de861649e785bb63859ad5eba5"
format_stamp: "Formatted at 2026-09-24 09:56:43 on dist-test-slave-w8d2"
server_key: "5642b5063709dcab08eb6c61760daf08"
server_key_iv: "bfdbbe4149afdd5d82a2db1a295bf7f9"
server_key_version: "encryptionkey@0"
I20260924 09:56:43.192636 8645 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790243735634835-388-0/minicluster-data/master-0/wal/instance:
uuid: "4ef3a6de861649e785bb63859ad5eba5"
format_stamp: "Formatted at 2026-09-24 09:56:43 on dist-test-slave-w8d2"
server_key: "5642b5063709dcab08eb6c61760daf08"
server_key_iv: "bfdbbe4149afdd5d82a2db1a295bf7f9"
server_key_version: "encryptionkey@0"
I20260924 09:56:43.197307 8645 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 09:56:43.200448 8660 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 09:56:43.201597 8645 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 09:56:43.201740 8645 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790243735634835-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790243735634835-388-0/minicluster-data/master-0/wal
uuid: "4ef3a6de861649e785bb63859ad5eba5"
format_stamp: "Formatted at 2026-09-24 09:56:43 on dist-test-slave-w8d2"
server_key: "5642b5063709dcab08eb6c61760daf08"
server_key_iv: "bfdbbe4149afdd5d82a2db1a295bf7f9"
server_key_version: "encryptionkey@0"
I20260924 09:56:43.201894 8645 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790243735634835-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790243735634835-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790243735634835-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 09:56:43.218611 8645 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 09:56:43.222405 8645 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 09:56:43.222628 8645 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 09:56:43.238111 8645 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:34237
I20260924 09:56:43.238132 8711 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:34237 every 8 connection(s)
I20260924 09:56:43.239394 8645 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790243735634835-388-0/minicluster-data/master-0/data/info.pb
I20260924 09:56:43.242750 8712 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 09:56:43.248971 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu as pid 8645
I20260924 09:56:43.249183 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790243735634835-388-0/minicluster-data/master-0/wal/instance
I20260924 09:56:43.249761 388 external_mini_cluster.cc:1428] Setting key 7c689f2c1d23f68122c1464b5c278522
I20260924 09:56:43.252347 8712 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ef3a6de861649e785bb63859ad5eba5: Bootstrap starting.
I20260924 09:56:43.255509 8712 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4ef3a6de861649e785bb63859ad5eba5: Neither blocks nor log segments found. Creating new log.
I20260924 09:56:43.256758 8712 log.cc:824] T 00000000000000000000000000000000 P 4ef3a6de861649e785bb63859ad5eba5: Log is configured to *not* fsync() on all Append() calls
I20260924 09:56:43.259810 8712 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ef3a6de861649e785bb63859ad5eba5: No bootstrap required, opened a new log
Sep 24 09:56:43 dist-test-slave-w8d2 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243802, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 09:56:43.267774 8712 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4ef3a6de861649e785bb63859ad5eba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ef3a6de861649e785bb63859ad5eba5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34237 } }
I20260924 09:56:43.268119 8712 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4ef3a6de861649e785bb63859ad5eba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 09:56:43.268200 8712 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4ef3a6de861649e785bb63859ad5eba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ef3a6de861649e785bb63859ad5eba5, State: Initialized, Role: FOLLOWER
I20260924 09:56:43.268805 8712 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4ef3a6de861649e785bb63859ad5eba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ef3a6de861649e785bb63859ad5eba5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34237 } }
I20260924 09:56:43.269068 8712 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4ef3a6de861649e785bb63859ad5eba5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 09:56:43.269198 8712 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4ef3a6de861649e785bb63859ad5eba5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 09:56:43.269367 8712 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4ef3a6de861649e785bb63859ad5eba5 [term 0 FOLLOWER]: Advancing to term 1
I20260924 09:56:43.271905 8715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:43.251664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48562 (local address 127.0.97.62:34237)
0924 09:56:43.252386 (+ 722us) server_negotiation.cc:206] Beginning negotiation
0924 09:56:43.252398 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 09:56:43.252438 (+ 40us) server_negotiation.cc:407] Connection header received
0924 09:56:43.254129 (+ 1691us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:56:43.254151 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:56:43.254531 (+ 380us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:56:43.254857 (+ 326us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:56:43.256611 (+ 1754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:43.258260 (+ 1649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:43.259959 (+ 1699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:43.260618 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:43.260739 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:43.264837 (+ 4098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:56:43.264845 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:56:43.264859 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:56:43.264898 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:56:43.267709 (+ 2811us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:56:43.268377 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:56:43.268384 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:56:43.268391 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:56:43.268589 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:56:43.268928 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:56:43.268933 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:56:43.268937 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:56:43.269618 (+ 681us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:56:43.269879 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0924 09:56:43.270194 (+ 315us) server_negotiation.cc:299] Negotiation successful
0924 09:56:43.270628 (+ 434us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":521,"thread_start_us":317,"threads_started":1}
I20260924 09:56:43.272387 8712 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4ef3a6de861649e785bb63859ad5eba5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ef3a6de861649e785bb63859ad5eba5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34237 } }
I20260924 09:56:43.272848 8712 leader_election.cc:302] T 00000000000000000000000000000000 P 4ef3a6de861649e785bb63859ad5eba5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4ef3a6de861649e785bb63859ad5eba5; no voters:
I20260924 09:56:43.273424 8712 leader_election.cc:288] T 00000000000000000000000000000000 P 4ef3a6de861649e785bb63859ad5eba5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 09:56:43.273524 8717 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4ef3a6de861649e785bb63859ad5eba5 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 09:56:43.274483 8717 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4ef3a6de861649e785bb63859ad5eba5 [term 1 LEADER]: Becoming Leader. State: Replica: 4ef3a6de861649e785bb63859ad5eba5, State: Running, Role: LEADER
I20260924 09:56:43.275043 8717 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4ef3a6de861649e785bb63859ad5eba5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ef3a6de861649e785bb63859ad5eba5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34237 } }
I20260924 09:56:43.275540 8712 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4ef3a6de861649e785bb63859ad5eba5 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 09:56:43.277640 8718 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ef3a6de861649e785bb63859ad5eba5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4ef3a6de861649e785bb63859ad5eba5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ef3a6de861649e785bb63859ad5eba5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34237 } } }
I20260924 09:56:43.277995 8718 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ef3a6de861649e785bb63859ad5eba5 [sys.catalog]: This master's current role is: LEADER
I20260924 09:56:43.278291 8719 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ef3a6de861649e785bb63859ad5eba5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4ef3a6de861649e785bb63859ad5eba5. Latest consensus state: current_term: 1 leader_uuid: "4ef3a6de861649e785bb63859ad5eba5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ef3a6de861649e785bb63859ad5eba5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34237 } } }
I20260924 09:56:43.278571 8719 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ef3a6de861649e785bb63859ad5eba5 [sys.catalog]: This master's current role is: LEADER
I20260924 09:56:43.280067 8726 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 09:56:43.284211 8726 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 09:56:43.291417 8726 catalog_manager.cc:1347] Generated new cluster ID: 26718839514d49e49fb3c9c61a3608ad
I20260924 09:56:43.291517 8726 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 09:56:43.302974 8726 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 09:56:43.304701 8726 catalog_manager.cc:1504] Loading token signing keys...
I20260924 09:56:43.313742 8726 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4ef3a6de861649e785bb63859ad5eba5: Generated new TSK 0
I20260924 09:56:43.314656 8726 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790243735634835-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790243735634835-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260924 09:56:43.368517 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
/tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790243735634835-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790243735634835-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790243735634835-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790243735634835-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34237
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790243735634835-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790243735634835-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790243735634835-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790243735634835-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 09:56:43.684585 8738 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 09:56:43.684860 8738 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 09:56:43.684993 8738 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 09:56:43.692022 8738 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 09:56:43.692127 8738 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 09:56:43.692269 8738 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260924 09:56:43.701514 8738 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790243735634835-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790243735634835-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790243735634835-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790243735634835-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34237
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790243735634835-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 09:56:43.702677 8738 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 09:56:43.703922 8738 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 09:56:43.716153 8743 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:56:43.716125 8744 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:56:43.716465 8746 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 09:56:43.717103 8738 server_base.cc:1034] running on GCE node
I20260924 09:56:43.717680 8738 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 09:56:43.718644 8738 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 09:56:43.719821 8738 hybrid_clock.cc:648] HybridClock initialized: now 1790243803719781 us; error 29 us; skew 500 ppm
I20260924 09:56:43.720095 8738 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 09:56:43 dist-test-slave-w8d2 krb5kdc[8632](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243803, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 09:56:43.726109 8738 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260924 09:56:43.727420 8738 webserver.cc:460] Webserver started at http://127.0.97.1:35945/ using document root <none> and password file <none>
I20260924 09:56:43.728027 8738 fs_manager.cc:359] Metadata directory not provided
I20260924 09:56:43.728132 8738 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 09:56:43.728379 8738 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 09:56:43.730999 8738 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790243735634835-388-0/minicluster-data/ts-0/data/instance:
uuid: "1cb4aff5f21d4ce6b4be7f07a81f5789"
format_stamp: "Formatted at 2026-09-24 09:56:43 on dist-test-slave-w8d2"
server_key: "aaae0dcd72708e2d98b0e19cf8682c73"
server_key_iv: "4366a45e04a214e6907b6f83cd1fe4ac"
server_key_version: "encryptionkey@0"
I20260924 09:56:43.731711 8738 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790243735634835-388-0/minicluster-data/ts-0/wal/instance:
uuid: "1cb4aff5f21d4ce6b4be7f07a81f5789"
format_stamp: "Formatted at 2026-09-24 09:56:43 on dist-test-slave-w8d2"
server_key: "aaae0dcd72708e2d98b0e19cf8682c73"
server_key_iv: "4366a45e04a214e6907b6f83cd1fe4ac"
server_key_version: "encryptionkey@0"
I20260924 09:56:43.736466 8738 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260924 09:56:43.739508 8753 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 09:56:43.740775 8738 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260924 09:56:43.740892 8738 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790243735634835-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790243735634835-388-0/minicluster-data/ts-0/wal
uuid: "1cb4aff5f21d4ce6b4be7f07a81f5789"
format_stamp: "Formatted at 2026-09-24 09:56:43 on dist-test-slave-w8d2"
server_key: "aaae0dcd72708e2d98b0e19cf8682c73"
server_key_iv: "4366a45e04a214e6907b6f83cd1fe4ac"
server_key_version: "encryptionkey@0"
I20260924 09:56:43.741048 8738 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790243735634835-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790243735634835-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790243735634835-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 09:56:43.758720 8738 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 09:56:43.762454 8738 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 09:56:43.762666 8738 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 09:56:43.763492 8738 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 09:56:43.764705 8738 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 09:56:43.764778 8738 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 09:56:43.764886 8738 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 09:56:43.764935 8738 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 09:56:43.792868 8738 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:34019
I20260924 09:56:43.792912 8865 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:34019 every 8 connection(s)
I20260924 09:56:43.794013 8738 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790243735634835-388-0/minicluster-data/ts-0/data/info.pb
I20260924 09:56:43.797204 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu as pid 8738
I20260924 09:56:43.797426 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790243735634835-388-0/minicluster-data/ts-0/wal/instance
I20260924 09:56:43.798225 388 external_mini_cluster.cc:1428] Setting key 808427e7585aa407b29acbb6d2420659
Sep 24 09:56:43 dist-test-slave-w8d2 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243803, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 09:56:43.815483 8868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:43.796448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:40411 (local address 127.0.97.62:34237)
0924 09:56:43.796851 (+ 403us) server_negotiation.cc:206] Beginning negotiation
0924 09:56:43.796856 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 09:56:43.800014 (+ 3158us) server_negotiation.cc:407] Connection header received
0924 09:56:43.800068 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:56:43.800073 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:56:43.800141 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:56:43.800250 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:56:43.801689 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:43.802676 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:43.805125 (+ 2449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:43.805723 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:43.805832 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:43.809938 (+ 4106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:56:43.809943 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:56:43.809947 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:56:43.809981 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:56:43.812332 (+ 2351us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:56:43.813144 (+ 812us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:56:43.813148 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:56:43.813151 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:56:43.813271 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:56:43.813676 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:56:43.813680 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:56:43.813682 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:56:43.814024 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:56:43.814151 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0924 09:56:43.814895 (+ 744us) server_negotiation.cc:299] Negotiation successful
0924 09:56:43.815111 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":212,"threads_started":1}
I20260924 09:56:43.816650 8869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:43.797030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34237 (local address 127.0.97.1:40411)
0924 09:56:43.797825 (+ 795us) negotiation.cc:107] Waiting for socket to connect
0924 09:56:43.797876 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0924 09:56:43.799585 (+ 1709us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:56:43.800471 (+ 886us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:56:43.800479 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:56:43.800963 (+ 484us) client_negotiation.cc:189] Negotiated authn=SASL
0924 09:56:43.801521 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:43.801537 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:43.802834 (+ 1297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:43.802840 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:43.804974 (+ 2134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:43.804993 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:43.805981 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:43.805989 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:43.806375 (+ 386us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:43.807063 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 09:56:43.807091 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0924 09:56:43.809659 (+ 2568us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 09:56:43.812476 (+ 2817us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 09:56:43.812485 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 09:56:43.812506 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0924 09:56:43.813020 (+ 514us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 09:56:43.813371 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 09:56:43.813377 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 09:56:43.813380 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 09:56:43.813561 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 09:56:43.814138 (+ 577us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 09:56:43.814146 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 09:56:43.814552 (+ 406us) client_negotiation.cc:769] Sending connection context
0924 09:56:43.814794 (+ 242us) client_negotiation.cc:240] Negotiation successful
0924 09:56:43.815369 (+ 575us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":564,"thread_start_us":314,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260924 09:56:43.818172 8866 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34237
I20260924 09:56:43.818456 8866 heartbeater.cc:464] Registering TS with master...
I20260924 09:56:43.819135 8866 heartbeater.cc:511] Master 127.0.97.62:34237 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260924 09:56:43.821183 8677 ts_manager.cc:194] Registered new tserver with Master: 1cb4aff5f21d4ce6b4be7f07a81f5789 (127.0.97.1:34019)
I20260924 09:56:43.822556 8677 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:40411
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790243735634835-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790243735634835-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260924 09:56:43.841423 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
/tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790243735634835-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790243735634835-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790243735634835-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790243735634835-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34237
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790243735634835-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790243735634835-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790243735634835-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790243735634835-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 09:56:44.142558 8872 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 09:56:44.142817 8872 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 09:56:44.142946 8872 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 09:56:44.149837 8872 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 09:56:44.149945 8872 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 09:56:44.150094 8872 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260924 09:56:44.159152 8872 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790243735634835-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790243735634835-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790243735634835-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790243735634835-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34237
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790243735634835-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 09:56:44.160352 8872 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 09:56:44.161571 8872 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 09:56:44.173516 8877 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:56:44.173484 8880 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:56:44.173436 8878 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 09:56:44.174253 8872 server_base.cc:1034] running on GCE node
I20260924 09:56:44.174798 8872 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 09:56:44.175670 8872 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 09:56:44.176862 8872 hybrid_clock.cc:648] HybridClock initialized: now 1790243804176805 us; error 52 us; skew 500 ppm
I20260924 09:56:44.177134 8872 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 09:56:44 dist-test-slave-w8d2 krb5kdc[8632](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243804, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 09:56:44.180022 8872 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260924 09:56:44.181245 8872 webserver.cc:460] Webserver started at http://127.0.97.2:41357/ using document root <none> and password file <none>
I20260924 09:56:44.181849 8872 fs_manager.cc:359] Metadata directory not provided
I20260924 09:56:44.181947 8872 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 09:56:44.182214 8872 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 09:56:44.184893 8872 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790243735634835-388-0/minicluster-data/ts-1/data/instance:
uuid: "a1edab552f3443108e9a26bd15a77205"
format_stamp: "Formatted at 2026-09-24 09:56:44 on dist-test-slave-w8d2"
server_key: "5b00e9e3cd7baaff9a49c863ef3bbe0d"
server_key_iv: "15028f150e4ac909374e99d24faf039e"
server_key_version: "encryptionkey@0"
I20260924 09:56:44.185581 8872 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790243735634835-388-0/minicluster-data/ts-1/wal/instance:
uuid: "a1edab552f3443108e9a26bd15a77205"
format_stamp: "Formatted at 2026-09-24 09:56:44 on dist-test-slave-w8d2"
server_key: "5b00e9e3cd7baaff9a49c863ef3bbe0d"
server_key_iv: "15028f150e4ac909374e99d24faf039e"
server_key_version: "encryptionkey@0"
I20260924 09:56:44.190371 8872 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.005s
I20260924 09:56:44.193526 8887 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 09:56:44.194732 8872 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 09:56:44.194876 8872 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790243735634835-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790243735634835-388-0/minicluster-data/ts-1/wal
uuid: "a1edab552f3443108e9a26bd15a77205"
format_stamp: "Formatted at 2026-09-24 09:56:44 on dist-test-slave-w8d2"
server_key: "5b00e9e3cd7baaff9a49c863ef3bbe0d"
server_key_iv: "15028f150e4ac909374e99d24faf039e"
server_key_version: "encryptionkey@0"
I20260924 09:56:44.195048 8872 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790243735634835-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790243735634835-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790243735634835-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 09:56:44.209295 8872 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 09:56:44.213016 8872 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 09:56:44.213249 8872 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 09:56:44.214054 8872 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 09:56:44.215220 8872 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 09:56:44.215310 8872 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 09:56:44.215413 8872 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 09:56:44.215466 8872 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 09:56:44.243440 8872 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:34931
I20260924 09:56:44.243490 8999 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:34931 every 8 connection(s)
I20260924 09:56:44.244719 8872 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790243735634835-388-0/minicluster-data/ts-1/data/info.pb
I20260924 09:56:44.249080 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu as pid 8872
I20260924 09:56:44.249245 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790243735634835-388-0/minicluster-data/ts-1/wal/instance
I20260924 09:56:44.249621 388 external_mini_cluster.cc:1428] Setting key 712ac3c9e75180d5b063e249c5119427
Sep 24 09:56:44 dist-test-slave-w8d2 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243804, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 09:56:44.265174 8868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:44.247170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:39559 (local address 127.0.97.62:34237)
0924 09:56:44.247379 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0924 09:56:44.247385 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 09:56:44.248580 (+ 1195us) server_negotiation.cc:407] Connection header received
0924 09:56:44.250996 (+ 2416us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:56:44.251001 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:56:44.251102 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:56:44.251172 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:56:44.253169 (+ 1997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:44.254112 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:44.256003 (+ 1891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:44.256539 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:44.256649 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:44.260152 (+ 3503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:56:44.260157 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:56:44.260160 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:56:44.260188 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:56:44.262492 (+ 2304us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:56:44.263202 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:56:44.263205 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:56:44.263208 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:56:44.263358 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:56:44.263741 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:56:44.263743 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:56:44.263745 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:56:44.263972 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:56:44.264070 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0924 09:56:44.264814 (+ 744us) server_negotiation.cc:299] Negotiation successful
0924 09:56:44.264968 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260924 09:56:44.266513 9002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:44.247657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34237 (local address 127.0.97.2:39559)
0924 09:56:44.248417 (+ 760us) negotiation.cc:107] Waiting for socket to connect
0924 09:56:44.248463 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0924 09:56:44.250702 (+ 2239us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:56:44.251685 (+ 983us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:56:44.251696 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:56:44.252336 (+ 640us) client_negotiation.cc:189] Negotiated authn=SASL
0924 09:56:44.253000 (+ 664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:44.253020 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:44.254266 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:44.254272 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:44.255885 (+ 1613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:44.255897 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:44.256709 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:44.256713 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:44.256892 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:44.257479 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 09:56:44.257506 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0924 09:56:44.259951 (+ 2445us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 09:56:44.262603 (+ 2652us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 09:56:44.262611 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 09:56:44.262624 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 09:56:44.263111 (+ 487us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 09:56:44.263441 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 09:56:44.263445 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 09:56:44.263448 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 09:56:44.263653 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 09:56:44.264076 (+ 423us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 09:56:44.264083 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 09:56:44.264444 (+ 361us) client_negotiation.cc:769] Sending connection context
0924 09:56:44.264701 (+ 257us) client_negotiation.cc:240] Negotiation successful
0924 09:56:44.265181 (+ 480us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":280,"threads_started":1}
I20260924 09:56:44.267903 9000 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34237
I20260924 09:56:44.268172 9000 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260924 09:56:44.268800 9000 heartbeater.cc:511] Master 127.0.97.62:34237 requested a full tablet report, sending...
I20260924 09:56:44.270071 8677 ts_manager.cc:194] Registered new tserver with Master: a1edab552f3443108e9a26bd15a77205 (127.0.97.2:34931)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260924 09:56:44.270898 8677 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:39559
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790243735634835-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790243735634835-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260924 09:56:44.292028 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
/tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790243735634835-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790243735634835-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790243735634835-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790243735634835-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34237
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790243735634835-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790243735634835-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790243735634835-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790243735634835-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 09:56:44.595642 9005 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 09:56:44.595906 9005 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 09:56:44.596026 9005 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 09:56:44.602982 9005 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 09:56:44.603081 9005 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 09:56:44.603226 9005 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260924 09:56:44.612591 9005 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790243735634835-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790243735634835-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790243735634835-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790243735634835-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34237
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790243735634835-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 09:56:44.613762 9005 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 09:56:44.614984 9005 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 09:56:44.626827 9013 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:56:44.626948 9011 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:56:44.626945 9010 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 09:56:44.627959 9005 server_base.cc:1034] running on GCE node
I20260924 09:56:44.628520 9005 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 09:56:44.629364 9005 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 09:56:44.630559 9005 hybrid_clock.cc:648] HybridClock initialized: now 1790243804630511 us; error 30 us; skew 500 ppm
I20260924 09:56:44.630863 9005 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 09:56:44 dist-test-slave-w8d2 krb5kdc[8632](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243804, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 09:56:44.633733 9005 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260924 09:56:44.635076 9005 webserver.cc:460] Webserver started at http://127.0.97.3:37767/ using document root <none> and password file <none>
I20260924 09:56:44.635751 9005 fs_manager.cc:359] Metadata directory not provided
I20260924 09:56:44.635864 9005 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 09:56:44.636175 9005 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 09:56:44.638875 9005 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790243735634835-388-0/minicluster-data/ts-2/data/instance:
uuid: "9f2c0c46fab14e9b93018b80e95d7be4"
format_stamp: "Formatted at 2026-09-24 09:56:44 on dist-test-slave-w8d2"
server_key: "53c67aff0644fc02c28ebe9856292bb4"
server_key_iv: "334267d7329bf7339a2b82b533386a5a"
server_key_version: "encryptionkey@0"
I20260924 09:56:44.639595 9005 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790243735634835-388-0/minicluster-data/ts-2/wal/instance:
uuid: "9f2c0c46fab14e9b93018b80e95d7be4"
format_stamp: "Formatted at 2026-09-24 09:56:44 on dist-test-slave-w8d2"
server_key: "53c67aff0644fc02c28ebe9856292bb4"
server_key_iv: "334267d7329bf7339a2b82b533386a5a"
server_key_version: "encryptionkey@0"
I20260924 09:56:44.644254 9005 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260924 09:56:44.647475 9020 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 09:56:44.648697 9005 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260924 09:56:44.648829 9005 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790243735634835-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790243735634835-388-0/minicluster-data/ts-2/wal
uuid: "9f2c0c46fab14e9b93018b80e95d7be4"
format_stamp: "Formatted at 2026-09-24 09:56:44 on dist-test-slave-w8d2"
server_key: "53c67aff0644fc02c28ebe9856292bb4"
server_key_iv: "334267d7329bf7339a2b82b533386a5a"
server_key_version: "encryptionkey@0"
I20260924 09:56:44.648998 9005 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790243735634835-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790243735634835-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790243735634835-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 09:56:44.671377 9005 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 09:56:44.674955 9005 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 09:56:44.675215 9005 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 09:56:44.676151 9005 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 09:56:44.677453 9005 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 09:56:44.677526 9005 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 09:56:44.677639 9005 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 09:56:44.677713 9005 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 09:56:44.706094 9005 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:46587
I20260924 09:56:44.706167 9132 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:46587 every 8 connection(s)
I20260924 09:56:44.707366 9005 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790243735634835-388-0/minicluster-data/ts-2/data/info.pb
I20260924 09:56:44.708927 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu as pid 9005
I20260924 09:56:44.709110 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790243735634835-388-0/minicluster-data/ts-2/wal/instance
I20260924 09:56:44.709522 388 external_mini_cluster.cc:1428] Setting key 79ec50d52c6ed628e8a494b27c03019e
Sep 24 09:56:44 dist-test-slave-w8d2 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243804, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 09:56:44.725446 8868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:44.709827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:47971 (local address 127.0.97.62:34237)
0924 09:56:44.710013 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0924 09:56:44.710017 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 09:56:44.711474 (+ 1457us) server_negotiation.cc:407] Connection header received
0924 09:56:44.713366 (+ 1892us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:56:44.713370 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:56:44.713435 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:56:44.713519 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:56:44.714815 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:44.715601 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:44.717037 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:44.717407 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:44.717497 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:44.720432 (+ 2935us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:56:44.720437 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:56:44.720440 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:56:44.720471 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:56:44.722615 (+ 2144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:56:44.723424 (+ 809us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:56:44.723429 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:56:44.723432 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:56:44.723563 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:56:44.724002 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:56:44.724006 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:56:44.724008 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:56:44.724339 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:56:44.724456 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0924 09:56:44.725049 (+ 593us) server_negotiation.cc:299] Negotiation successful
0924 09:56:44.725201 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 09:56:44.726341 9135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:44.710485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34237 (local address 127.0.97.3:47971)
0924 09:56:44.711322 (+ 837us) negotiation.cc:107] Waiting for socket to connect
0924 09:56:44.711378 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0924 09:56:44.713119 (+ 1741us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:56:44.713676 (+ 557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:56:44.713683 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:56:44.714141 (+ 458us) client_negotiation.cc:189] Negotiated authn=SASL
0924 09:56:44.714672 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:44.714689 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:44.715717 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:44.715721 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:44.716912 (+ 1191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:44.716923 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:44.717563 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:44.717568 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:44.717744 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:44.718285 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 09:56:44.718305 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 09:56:44.720258 (+ 1953us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 09:56:44.722752 (+ 2494us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 09:56:44.722759 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 09:56:44.722770 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0924 09:56:44.723321 (+ 551us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 09:56:44.723663 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 09:56:44.723668 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 09:56:44.723671 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 09:56:44.723894 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 09:56:44.724459 (+ 565us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 09:56:44.724465 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 09:56:44.724756 (+ 291us) client_negotiation.cc:769] Sending connection context
0924 09:56:44.724969 (+ 213us) client_negotiation.cc:240] Negotiation successful
0924 09:56:44.725330 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":546,"thread_start_us":339,"threads_started":1}
I20260924 09:56:44.727568 9133 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34237
I20260924 09:56:44.727820 9133 heartbeater.cc:464] Registering TS with master...
I20260924 09:56:44.728470 9133 heartbeater.cc:511] Master 127.0.97.62:34237 requested a full tablet report, sending...
I20260924 09:56:44.729717 8677 ts_manager.cc:194] Registered new tserver with Master: 9f2c0c46fab14e9b93018b80e95d7be4 (127.0.97.3:46587)
I20260924 09:56:44.730448 8677 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:47971
I20260924 09:56:44.736234 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790243735634835-388-0/krb5kdc/oryx_127.0.97.1.keytab.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790243735634835-388-0/krb5kdc/oryx_127.0.97.1.keytab.
WARNING: no policy specified for oryx/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790243735634835-388-0/krb5kdc/oryx_127.0.97.2.keytab.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790243735634835-388-0/krb5kdc/oryx_127.0.97.2.keytab.
I20260924 09:56:44.825259 8866 heartbeater.cc:503] Master 127.0.97.62:34237 was elected leader, sending a full tablet report...
WARNING: no policy specified for oryx/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790243735634835-388-0/krb5kdc/oryx_127.0.97.3.keytab.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790243735634835-388-0/krb5kdc/oryx_127.0.97.3.keytab.
I20260924 09:56:44.855625 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu pid 8738
Sep 24 09:56:44 dist-test-slave-w8d2 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243802, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260924 09:56:44.870152 9143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:44.856749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49370 (local address 127.0.97.1:34019)
0924 09:56:44.857401 (+ 652us) server_negotiation.cc:206] Beginning negotiation
0924 09:56:44.857408 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 09:56:44.857433 (+ 25us) server_negotiation.cc:407] Connection header received
0924 09:56:44.857492 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:56:44.857498 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:56:44.857706 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:56:44.857842 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:56:44.858991 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:44.860025 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:44.861137 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:44.861662 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:44.861749 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:44.865306 (+ 3557us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:56:44.865319 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:56:44.865339 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:56:44.865376 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:56:44.867976 (+ 2600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:56:44.868535 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:56:44.868542 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:56:44.868545 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:56:44.868674 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:56:44.868926 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:56:44.868929 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:56:44.868931 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:56:44.869217 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:56:44.869423 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0924 09:56:44.869611 (+ 188us) server_negotiation.cc:299] Negotiation successful
0924 09:56:44.869783 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":493,"thread_start_us":350,"threads_started":1}
I20260924 09:56:44.871390 8861 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 09:56:44.951560 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu with pid 8738
I20260924 09:56:44.964470 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu pid 8872
Sep 24 09:56:44 dist-test-slave-w8d2 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243802, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260924 09:56:44.979405 9145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:44.965258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47684 (local address 127.0.97.2:34931)
0924 09:56:44.965844 (+ 586us) server_negotiation.cc:206] Beginning negotiation
0924 09:56:44.965851 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 09:56:44.965877 (+ 26us) server_negotiation.cc:407] Connection header received
0924 09:56:44.965938 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:56:44.965943 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:56:44.966152 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:56:44.966281 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:56:44.967465 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:44.968455 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:44.969774 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:44.970302 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:44.970394 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:44.974068 (+ 3674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:56:44.974080 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:56:44.974094 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:56:44.974127 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:56:44.976785 (+ 2658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:56:44.977430 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:56:44.977440 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:56:44.977442 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:56:44.977563 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:56:44.977886 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:56:44.977890 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:56:44.977892 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:56:44.978214 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:56:44.978489 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0924 09:56:44.978793 (+ 304us) server_negotiation.cc:299] Negotiation successful
0924 09:56:44.979043 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":447,"thread_start_us":308,"threads_started":1}
I20260924 09:56:44.980630 8995 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 09:56:45.060009 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu with pid 8872
I20260924 09:56:45.072657 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu pid 9005
Sep 24 09:56:45 dist-test-slave-w8d2 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243802, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260924 09:56:45.087590 9147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:45.073573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44592 (local address 127.0.97.3:46587)
0924 09:56:45.074159 (+ 586us) server_negotiation.cc:206] Beginning negotiation
0924 09:56:45.074166 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 09:56:45.074192 (+ 26us) server_negotiation.cc:407] Connection header received
0924 09:56:45.074248 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:56:45.074254 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:56:45.074482 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:56:45.074603 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:56:45.075855 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:45.076910 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:45.078095 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:45.078669 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:45.078792 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:45.082739 (+ 3947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:56:45.082750 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:56:45.082764 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:56:45.082796 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:56:45.085279 (+ 2483us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:56:45.085867 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:56:45.085873 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:56:45.085875 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:56:45.085987 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:56:45.086248 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:56:45.086250 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:56:45.086252 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:56:45.086587 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:56:45.086805 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0924 09:56:45.087028 (+ 223us) server_negotiation.cc:299] Negotiation successful
0924 09:56:45.087215 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":446,"thread_start_us":301,"threads_started":1}
I20260924 09:56:45.088798 9128 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 09:56:45.168610 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu with pid 9005
I20260924 09:56:45.180851 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu pid 8645
I20260924 09:56:45.181891 8707 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 09:56:45.281142 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu with pid 8645
I20260924 09:56:45.297217 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
/tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790243735634835-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790243735634835-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790243735634835-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790243735634835-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34237
--webserver_interface=127.0.97.62
--webserver_port=41573
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790243735634835-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:34237
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790243735634835-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790243735634835-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790243735634835-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 09:56:45.600128 9150 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 09:56:45.600423 9150 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 09:56:45.600556 9150 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 09:56:45.607627 9150 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 09:56:45.607738 9150 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 09:56:45.607815 9150 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 09:56:45.607885 9150 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 09:56:45.607940 9150 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 09:56:45.617851 9150 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790243735634835-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790243735634835-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:34237
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790243735634835-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34237
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790243735634835-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=41573
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790243735634835-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 09:56:45.619109 9150 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 09:56:45.620402 9150 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 09:56:45.630967 9155 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:56:45.631000 9158 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:56:45.630941 9156 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 09:56:45.631773 9150 server_base.cc:1034] running on GCE node
I20260924 09:56:45.632489 9150 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 09:56:45.633744 9150 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 09:56:45.634925 9150 hybrid_clock.cc:648] HybridClock initialized: now 1790243805634866 us; error 46 us; skew 500 ppm
I20260924 09:56:45.635227 9150 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 09:56:45 dist-test-slave-w8d2 krb5kdc[8632](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243805, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 09:56:45.637971 9150 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260924 09:56:45.639254 9150 webserver.cc:460] Webserver started at http://127.0.97.62:41573/ using document root <none> and password file <none>
I20260924 09:56:45.639850 9150 fs_manager.cc:359] Metadata directory not provided
I20260924 09:56:45.639950 9150 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 09:56:45.644753 9150 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.002s
I20260924 09:56:45.647149 9165 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 09:56:45.648355 9150 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260924 09:56:45.648470 9150 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790243735634835-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790243735634835-388-0/minicluster-data/master-0/wal
uuid: "4ef3a6de861649e785bb63859ad5eba5"
format_stamp: "Formatted at 2026-09-24 09:56:43 on dist-test-slave-w8d2"
server_key: "5642b5063709dcab08eb6c61760daf08"
server_key_iv: "bfdbbe4149afdd5d82a2db1a295bf7f9"
server_key_version: "encryptionkey@0"
I20260924 09:56:45.649323 9150 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790243735634835-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790243735634835-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790243735634835-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 09:56:45.667814 9150 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 09:56:45.671455 9150 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 09:56:45.671659 9150 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 09:56:45.687556 9150 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:34237
I20260924 09:56:45.687587 9216 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:34237 every 8 connection(s)
I20260924 09:56:45.688961 9150 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790243735634835-388-0/minicluster-data/master-0/data/info.pb
I20260924 09:56:45.695526 9217 sys_catalog.cc:263] Verifying existing consensus state
I20260924 09:56:45.696973 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu as pid 9150
I20260924 09:56:45.697863 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
/tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790243735634835-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790243735634835-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790243735634835-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790243735634835-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:34019
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:34237
--webserver_port=35945
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790243735634835-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.97.1
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790243735634835-388-0/krb5kdc/oryx_127.0.97.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790243735634835-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790243735634835-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790243735634835-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 09:56:45.699616 9217 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ef3a6de861649e785bb63859ad5eba5: Bootstrap starting.
I20260924 09:56:45.719305 9217 log.cc:824] T 00000000000000000000000000000000 P 4ef3a6de861649e785bb63859ad5eba5: Log is configured to *not* fsync() on all Append() calls
I20260924 09:56:45.727936 9217 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ef3a6de861649e785bb63859ad5eba5: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 09:56:45.728439 9217 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ef3a6de861649e785bb63859ad5eba5: Bootstrap complete.
I20260924 09:56:45.737136 9217 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4ef3a6de861649e785bb63859ad5eba5 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ef3a6de861649e785bb63859ad5eba5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34237 } }
I20260924 09:56:45.737951 9217 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4ef3a6de861649e785bb63859ad5eba5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ef3a6de861649e785bb63859ad5eba5, State: Initialized, Role: FOLLOWER
I20260924 09:56:45.738710 9217 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4ef3a6de861649e785bb63859ad5eba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ef3a6de861649e785bb63859ad5eba5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34237 } }
I20260924 09:56:45.738961 9217 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4ef3a6de861649e785bb63859ad5eba5 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 09:56:45.739090 9217 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4ef3a6de861649e785bb63859ad5eba5 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 09:56:45.739283 9217 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4ef3a6de861649e785bb63859ad5eba5 [term 1 FOLLOWER]: Advancing to term 2
I20260924 09:56:45.743065 9217 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4ef3a6de861649e785bb63859ad5eba5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ef3a6de861649e785bb63859ad5eba5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34237 } }
I20260924 09:56:45.743598 9217 leader_election.cc:302] T 00000000000000000000000000000000 P 4ef3a6de861649e785bb63859ad5eba5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4ef3a6de861649e785bb63859ad5eba5; no voters:
I20260924 09:56:45.744169 9217 leader_election.cc:288] T 00000000000000000000000000000000 P 4ef3a6de861649e785bb63859ad5eba5 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 09:56:45.744300 9221 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4ef3a6de861649e785bb63859ad5eba5 [term 2 FOLLOWER]: Leader election won for term 2
I20260924 09:56:45.745406 9221 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4ef3a6de861649e785bb63859ad5eba5 [term 2 LEADER]: Becoming Leader. State: Replica: 4ef3a6de861649e785bb63859ad5eba5, State: Running, Role: LEADER
I20260924 09:56:45.746019 9221 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4ef3a6de861649e785bb63859ad5eba5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ef3a6de861649e785bb63859ad5eba5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34237 } }
I20260924 09:56:45.746300 9217 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4ef3a6de861649e785bb63859ad5eba5 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 09:56:45.748680 9223 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ef3a6de861649e785bb63859ad5eba5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4ef3a6de861649e785bb63859ad5eba5. Latest consensus state: current_term: 2 leader_uuid: "4ef3a6de861649e785bb63859ad5eba5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ef3a6de861649e785bb63859ad5eba5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34237 } } }
I20260924 09:56:45.749056 9223 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ef3a6de861649e785bb63859ad5eba5 [sys.catalog]: This master's current role is: LEADER
I20260924 09:56:45.748672 9222 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ef3a6de861649e785bb63859ad5eba5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "4ef3a6de861649e785bb63859ad5eba5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ef3a6de861649e785bb63859ad5eba5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34237 } } }
I20260924 09:56:45.749420 9222 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ef3a6de861649e785bb63859ad5eba5 [sys.catalog]: This master's current role is: LEADER
I20260924 09:56:45.759083 9228 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 09:56:45.765328 9228 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 09:56:45.767096 9228 catalog_manager.cc:1259] Loaded cluster ID: 26718839514d49e49fb3c9c61a3608ad
I20260924 09:56:45.767253 9228 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 09:56:45.771334 9228 catalog_manager.cc:1504] Loading token signing keys...
I20260924 09:56:45.772923 9228 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 4ef3a6de861649e785bb63859ad5eba5: Loaded TSK: 0
I20260924 09:56:45.774653 9228 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 09:56:46.033825 9219 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 09:56:46.034099 9219 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 09:56:46.034228 9219 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 09:56:46.041229 9219 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 09:56:46.041345 9219 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 09:56:46.041495 9219 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260924 09:56:46.051080 9219 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790243735634835-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790243735634835-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790243735634835-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:34019
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790243735634835-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=35945
--tserver_master_addrs=127.0.97.62:34237
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790243735634835-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 09:56:46.052448 9219 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 09:56:46.053718 9219 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 09:56:46.067109 9243 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:56:46.067137 9244 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:56:46.067898 9246 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 09:56:46.068240 9219 server_base.cc:1034] running on GCE node
I20260924 09:56:46.068956 9219 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 09:56:46.070050 9219 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 09:56:46.071300 9219 hybrid_clock.cc:648] HybridClock initialized: now 1790243806071239 us; error 45 us; skew 500 ppm
I20260924 09:56:46.071674 9219 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 09:56:46 dist-test-slave-w8d2 krb5kdc[8632](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243806, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 09:56:46.075362 9219 init.cc:377] Logged in from keytab as oryx/127.0.97.1@KRBTEST.COM (short username oryx)
I20260924 09:56:46.076963 9219 webserver.cc:460] Webserver started at http://127.0.97.1:35945/ using document root <none> and password file <none>
I20260924 09:56:46.077809 9219 fs_manager.cc:359] Metadata directory not provided
I20260924 09:56:46.077956 9219 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 09:56:46.084393 9219 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 09:56:46.087491 9253 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 09:56:46.089076 9219 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260924 09:56:46.089232 9219 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790243735634835-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790243735634835-388-0/minicluster-data/ts-0/wal
uuid: "1cb4aff5f21d4ce6b4be7f07a81f5789"
format_stamp: "Formatted at 2026-09-24 09:56:43 on dist-test-slave-w8d2"
server_key: "aaae0dcd72708e2d98b0e19cf8682c73"
server_key_iv: "4366a45e04a214e6907b6f83cd1fe4ac"
server_key_version: "encryptionkey@0"
I20260924 09:56:46.090466 9219 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790243735634835-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790243735634835-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790243735634835-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 09:56:46.116070 9219 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 09:56:46.120949 9219 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 09:56:46.121207 9219 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 09:56:46.122278 9219 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 09:56:46.123857 9219 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 09:56:46.123967 9219 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 09:56:46.124064 9219 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 09:56:46.124130 9219 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 09:56:46.152293 9219 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:34019
I20260924 09:56:46.152357 9365 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:34019 every 8 connection(s)
I20260924 09:56:46.153690 9219 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790243735634835-388-0/minicluster-data/ts-0/data/info.pb
I20260924 09:56:46.160885 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu as pid 9219
I20260924 09:56:46.161406 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
/tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790243735634835-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790243735634835-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790243735634835-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790243735634835-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:34931
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:34237
--webserver_port=41357
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790243735634835-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.97.2
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790243735634835-388-0/krb5kdc/oryx_127.0.97.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790243735634835-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790243735634835-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790243735634835-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 09:56:46 dist-test-slave-w8d2 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 09:56:46 dist-test-slave-w8d2 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260924 09:56:46.170923 9368 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 09:56:46.156387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:41555 (local address 127.0.97.62:34237)
0924 09:56:46.157083 (+ 696us) server_negotiation.cc:206] Beginning negotiation
0924 09:56:46.157098 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 09:56:46.157342 (+ 244us) server_negotiation.cc:407] Connection header received
0924 09:56:46.159387 (+ 2045us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:56:46.159400 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:56:46.159750 (+ 350us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:56:46.160071 (+ 321us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:56:46.161563 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:46.162816 (+ 1253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:46.164341 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:46.164976 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:46.165144 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:46.170111 (+ 4967us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:41555: BlockingRecv error: recv got EOF from 127.0.97.1:41555 (error 108)
Metrics: {"server-negotiator.queue_time_us":436,"thread_start_us":210,"threads_started":1}
I20260924 09:56:46.170943 9369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:46.156449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34237 (local address 127.0.97.1:41555)
0924 09:56:46.157189 (+ 740us) negotiation.cc:107] Waiting for socket to connect
0924 09:56:46.157224 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 09:56:46.158827 (+ 1603us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:56:46.160233 (+ 1406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:56:46.160245 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:56:46.160834 (+ 589us) client_negotiation.cc:189] Negotiated authn=SASL
0924 09:56:46.161383 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:46.161402 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:46.162977 (+ 1575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:46.162981 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:46.164211 (+ 1230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:46.164223 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:46.165199 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:46.165206 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:46.165438 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:46.166071 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 09:56:46.166096 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0924 09:56:46.169809 (+ 3713us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:34237: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":312,"threads_started":1}
W20260924 09:56:46.171610 9366 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:34237 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.97.62:34237: Client connection negotiation failed: client connection to 127.0.97.62:34237: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Sep 24 09:56:46 dist-test-slave-w8d2 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 09:56:46 dist-test-slave-w8d2 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 09:56:46.179661 9369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:46.172152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34237 (local address 127.0.97.1:55043)
0924 09:56:46.172308 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0924 09:56:46.172326 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 09:56:46.172415 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:56:46.172716 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:56:46.172719 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:56:46.172999 (+ 280us) client_negotiation.cc:189] Negotiated authn=SASL
0924 09:56:46.173130 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:46.173138 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:46.174407 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:46.174411 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:46.175406 (+ 995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:46.175417 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:46.176250 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:46.176253 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:46.176363 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:46.176851 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 09:56:46.176867 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 09:56:46.179461 (+ 2594us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:34237: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":46}
W20260924 09:56:46.179925 9368 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 09:56:46.172287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:55043 (local address 127.0.97.62:34237)
0924 09:56:46.172436 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0924 09:56:46.172441 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 09:56:46.172477 (+ 36us) server_negotiation.cc:407] Connection header received
0924 09:56:46.172527 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:56:46.172531 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:56:46.172608 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:56:46.172689 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:56:46.173245 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:46.174267 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:46.175532 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:46.176130 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:46.176248 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:46.179705 (+ 3457us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:55043: BlockingRecv error: recv got EOF from 127.0.97.1:55043 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Sep 24 09:56:46 dist-test-slave-w8d2 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 09:56:46 dist-test-slave-w8d2 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 09:56:46.187187 9369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:46.180301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34237 (local address 127.0.97.1:59045)
0924 09:56:46.180436 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0924 09:56:46.180454 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 09:56:46.180542 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:56:46.180891 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:56:46.180896 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:56:46.181163 (+ 267us) client_negotiation.cc:189] Negotiated authn=SASL
0924 09:56:46.181290 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:46.181298 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:46.182213 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:46.182217 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:46.183084 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:46.183095 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:46.183793 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:46.183798 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:46.183940 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:46.184440 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 09:56:46.184456 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 09:56:46.186913 (+ 2457us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:34237: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":31}
W20260924 09:56:46.187415 9368 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 09:56:46.180512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:59045 (local address 127.0.97.62:34237)
0924 09:56:46.180658 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0924 09:56:46.180663 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 09:56:46.180677 (+ 14us) server_negotiation.cc:407] Connection header received
0924 09:56:46.180715 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:56:46.180718 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:56:46.180778 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:56:46.180854 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:56:46.181410 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:46.182091 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:46.183202 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:46.183669 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:46.183771 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:46.187141 (+ 3370us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:59045: BlockingRecv error: recv got EOF from 127.0.97.1:59045 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260924 09:56:46.187489 9366 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 09:56:46.469213 9370 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 09:56:46.469504 9370 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 09:56:46.469638 9370 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 09:56:46.476723 9370 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 09:56:46.476838 9370 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 09:56:46.476989 9370 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260924 09:56:46.486450 9370 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790243735634835-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790243735634835-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790243735634835-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:34931
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790243735634835-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=41357
--tserver_master_addrs=127.0.97.62:34237
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790243735634835-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 09:56:46.487797 9370 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 09:56:46.489063 9370 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 09:56:46.501353 9375 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:56:46.501315 9378 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:56:46.501394 9376 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 09:56:46.504459 9370 server_base.cc:1034] running on GCE node
I20260924 09:56:46.505036 9370 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 09:56:46.505927 9370 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 09:56:46.507112 9370 hybrid_clock.cc:648] HybridClock initialized: now 1790243806507062 us; error 42 us; skew 500 ppm
I20260924 09:56:46.507432 9370 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 09:56:46 dist-test-slave-w8d2 krb5kdc[8632](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243806, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 09:56:46.510183 9370 init.cc:377] Logged in from keytab as oryx/127.0.97.2@KRBTEST.COM (short username oryx)
I20260924 09:56:46.511564 9370 webserver.cc:460] Webserver started at http://127.0.97.2:41357/ using document root <none> and password file <none>
I20260924 09:56:46.512202 9370 fs_manager.cc:359] Metadata directory not provided
I20260924 09:56:46.512305 9370 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 09:56:46.517244 9370 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260924 09:56:46.519673 9385 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 09:56:46.520969 9370 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260924 09:56:46.521103 9370 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790243735634835-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790243735634835-388-0/minicluster-data/ts-1/wal
uuid: "a1edab552f3443108e9a26bd15a77205"
format_stamp: "Formatted at 2026-09-24 09:56:44 on dist-test-slave-w8d2"
server_key: "5b00e9e3cd7baaff9a49c863ef3bbe0d"
server_key_iv: "15028f150e4ac909374e99d24faf039e"
server_key_version: "encryptionkey@0"
I20260924 09:56:46.521968 9370 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790243735634835-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790243735634835-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790243735634835-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 09:56:46.539652 9370 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 09:56:46.543205 9370 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 09:56:46.543457 9370 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 09:56:46.544327 9370 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 09:56:46.545507 9370 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 09:56:46.545601 9370 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 09:56:46.545702 9370 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 09:56:46.545768 9370 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 09:56:46.574018 9370 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:34931
I20260924 09:56:46.574069 9497 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:34931 every 8 connection(s)
I20260924 09:56:46.575402 9370 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790243735634835-388-0/minicluster-data/ts-1/data/info.pb
I20260924 09:56:46.581893 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu as pid 9370
I20260924 09:56:46.582576 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
/tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790243735634835-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790243735634835-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790243735634835-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790243735634835-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:46587
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:34237
--webserver_port=37767
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790243735634835-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.97.3
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790243735634835-388-0/krb5kdc/oryx_127.0.97.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790243735634835-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790243735634835-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790243735634835-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 09:56:46 dist-test-slave-w8d2 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 09:56:46 dist-test-slave-w8d2 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260924 09:56:46.591113 9368 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 09:56:46.577659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:59779 (local address 127.0.97.62:34237)
0924 09:56:46.577871 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0924 09:56:46.577877 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 09:56:46.579074 (+ 1197us) server_negotiation.cc:407] Connection header received
0924 09:56:46.580788 (+ 1714us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:56:46.580792 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:56:46.580856 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:56:46.580933 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:56:46.582653 (+ 1720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:46.583405 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:46.585385 (+ 1980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:46.585790 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:46.585875 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:46.590933 (+ 5058us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:59779: BlockingRecv error: recv got EOF from 127.0.97.2:59779 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
I20260924 09:56:46.592237 9500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:46.578230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34237 (local address 127.0.97.2:59779)
0924 09:56:46.578911 (+ 681us) negotiation.cc:107] Waiting for socket to connect
0924 09:56:46.578949 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 09:56:46.580542 (+ 1593us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:56:46.581123 (+ 581us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:56:46.581130 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:56:46.581740 (+ 610us) client_negotiation.cc:189] Negotiated authn=SASL
0924 09:56:46.582470 (+ 730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:46.582506 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:46.583564 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:46.583570 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:46.585272 (+ 1702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:46.585285 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:46.585992 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:46.585998 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:46.586251 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:46.586909 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 09:56:46.586935 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0924 09:56:46.590781 (+ 3846us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:34237: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":279,"threads_started":1}
W20260924 09:56:46.592944 9498 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:34237 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.97.62:34237: Client connection negotiation failed: client connection to 127.0.97.62:34237: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Sep 24 09:56:46 dist-test-slave-w8d2 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 09:56:46 dist-test-slave-w8d2 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 09:56:46.601320 9500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:46.593447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34237 (local address 127.0.97.2:46695)
0924 09:56:46.593596 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0924 09:56:46.593612 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 09:56:46.593714 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:56:46.593977 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:56:46.593980 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:56:46.594337 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0924 09:56:46.594487 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:46.594497 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:46.595509 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:46.595513 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:46.596807 (+ 1294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:46.596820 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:46.597413 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:46.597418 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:46.597576 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:46.598078 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 09:56:46.598097 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 09:56:46.601053 (+ 2956us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:34237: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":40}
W20260924 09:56:46.601369 9368 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 09:56:46.593612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:46695 (local address 127.0.97.62:34237)
0924 09:56:46.593749 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0924 09:56:46.593753 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 09:56:46.593768 (+ 15us) server_negotiation.cc:407] Connection header received
0924 09:56:46.593808 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:56:46.593812 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:56:46.593862 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:56:46.593943 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:56:46.594602 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:46.595381 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:46.596918 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:46.597288 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:46.597379 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:46.601211 (+ 3832us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:46695: BlockingRecv error: recv got EOF from 127.0.97.2:46695 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 24 09:56:46 dist-test-slave-w8d2 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 09:56:46 dist-test-slave-w8d2 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 09:56:46.608771 9500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:46.601981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34237 (local address 127.0.97.2:49437)
0924 09:56:46.602161 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0924 09:56:46.602179 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 09:56:46.602274 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:56:46.602527 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:56:46.602532 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:56:46.602800 (+ 268us) client_negotiation.cc:189] Negotiated authn=SASL
0924 09:56:46.602923 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:46.602930 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:46.603862 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:46.603867 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:46.604737 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:46.604747 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:46.605348 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:46.605353 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:46.605503 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:46.605984 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 09:56:46.605999 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 09:56:46.608490 (+ 2491us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:34237: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":49}
W20260924 09:56:46.608894 9368 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 09:56:46.602019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:49437 (local address 127.0.97.62:34237)
0924 09:56:46.602161 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0924 09:56:46.602166 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 09:56:46.602223 (+ 57us) server_negotiation.cc:407] Connection header received
0924 09:56:46.602364 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:56:46.602369 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:56:46.602424 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:56:46.602496 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:56:46.603039 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:46.603748 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:46.604854 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:46.605230 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:46.605328 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:46.608681 (+ 3353us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:49437: BlockingRecv error: recv got EOF from 127.0.97.2:49437 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260924 09:56:46.609174 9498 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 09:56:46.894399 9501 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 09:56:46.894655 9501 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 09:56:46.894769 9501 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 09:56:46.901688 9501 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 09:56:46.901784 9501 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 09:56:46.901929 9501 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260924 09:56:46.911134 9501 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790243735634835-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790243735634835-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790243735634835-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:46587
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790243735634835-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=37767
--tserver_master_addrs=127.0.97.62:34237
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790243735634835-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 09:56:46.912389 9501 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 09:56:46.913599 9501 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 09:56:46.925559 9506 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:56:46.925525 9507 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:56:46.926020 9509 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 09:56:46.926381 9501 server_base.cc:1034] running on GCE node
I20260924 09:56:46.926929 9501 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 09:56:46.927817 9501 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 09:56:46.929005 9501 hybrid_clock.cc:648] HybridClock initialized: now 1790243806928958 us; error 40 us; skew 500 ppm
I20260924 09:56:46.929287 9501 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 09:56:46 dist-test-slave-w8d2 krb5kdc[8632](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243806, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 09:56:46.932103 9501 init.cc:377] Logged in from keytab as oryx/127.0.97.3@KRBTEST.COM (short username oryx)
I20260924 09:56:46.933348 9501 webserver.cc:460] Webserver started at http://127.0.97.3:37767/ using document root <none> and password file <none>
I20260924 09:56:46.933952 9501 fs_manager.cc:359] Metadata directory not provided
I20260924 09:56:46.934056 9501 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 09:56:46.938978 9501 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260924 09:56:46.941418 9516 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 09:56:46.942698 9501 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260924 09:56:46.942834 9501 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790243735634835-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790243735634835-388-0/minicluster-data/ts-2/wal
uuid: "9f2c0c46fab14e9b93018b80e95d7be4"
format_stamp: "Formatted at 2026-09-24 09:56:44 on dist-test-slave-w8d2"
server_key: "53c67aff0644fc02c28ebe9856292bb4"
server_key_iv: "334267d7329bf7339a2b82b533386a5a"
server_key_version: "encryptionkey@0"
I20260924 09:56:46.943768 9501 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790243735634835-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790243735634835-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790243735634835-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 09:56:46.959928 9501 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 09:56:46.963678 9501 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 09:56:46.963912 9501 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 09:56:46.964829 9501 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 09:56:46.966013 9501 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 09:56:46.966106 9501 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 09:56:46.966209 9501 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 09:56:46.966272 9501 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 09:56:46.994891 9501 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:46587
I20260924 09:56:46.994971 9628 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:46587 every 8 connection(s)
I20260924 09:56:46.996446 9501 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790243735634835-388-0/minicluster-data/ts-2/data/info.pb
I20260924 09:56:47.004336 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu as pid 9501
Sep 24 09:56:47 dist-test-slave-w8d2 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 09:56:47 dist-test-slave-w8d2 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260924 09:56:47.012074 9368 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 09:56:46.998838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:59803 (local address 127.0.97.62:34237)
0924 09:56:46.998996 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0924 09:56:46.999001 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 09:56:47.000194 (+ 1193us) server_negotiation.cc:407] Connection header received
0924 09:56:47.001899 (+ 1705us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:56:47.001905 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:56:47.001982 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:56:47.002064 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:56:47.003553 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:47.004260 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:47.006109 (+ 1849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:47.006538 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:47.006639 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:47.011895 (+ 5256us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:59803: BlockingRecv error: recv got EOF from 127.0.97.3:59803 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 09:56:47.013079 9631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:46.999378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34237 (local address 127.0.97.3:59803)
0924 09:56:47.000048 (+ 670us) negotiation.cc:107] Waiting for socket to connect
0924 09:56:47.000085 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 09:56:47.001648 (+ 1563us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:56:47.002310 (+ 662us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:56:47.002318 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:56:47.002813 (+ 495us) client_negotiation.cc:189] Negotiated authn=SASL
0924 09:56:47.003400 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:47.003420 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:47.004402 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:47.004406 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:47.005989 (+ 1583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:47.006000 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:47.006741 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:47.006749 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:47.006975 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:47.007698 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 09:56:47.007728 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0924 09:56:47.011764 (+ 4036us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:34237: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":269,"threads_started":1}
W20260924 09:56:47.013734 9629 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:34237 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.97.62:34237: Client connection negotiation failed: client connection to 127.0.97.62:34237: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
I20260924 09:56:47.019537 9633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:47.005256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48564 (local address 127.0.97.62:34237)
0924 09:56:47.005892 (+ 636us) server_negotiation.cc:206] Beginning negotiation
0924 09:56:47.005896 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 09:56:47.005911 (+ 15us) server_negotiation.cc:407] Connection header received
0924 09:56:47.005950 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:56:47.005954 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:56:47.006004 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:56:47.006103 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:56:47.007524 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:47.008607 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:47.010039 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:47.010452 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:47.010536 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:47.013462 (+ 2926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:56:47.013473 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:56:47.013492 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:56:47.013539 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:56:47.016671 (+ 3132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:56:47.017349 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:56:47.017355 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:56:47.017360 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:56:47.017513 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:56:47.017850 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:56:47.017853 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:56:47.017855 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:56:47.018313 (+ 458us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:56:47.018551 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0924 09:56:47.018927 (+ 376us) server_negotiation.cc:299] Negotiation successful
0924 09:56:47.019275 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":555,"thread_start_us":455,"threads_started":1}
Sep 24 09:56:47 dist-test-slave-w8d2 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 09:56:47 dist-test-slave-w8d2 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 09:56:47.021654 9631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:47.014249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34237 (local address 127.0.97.3:59877)
0924 09:56:47.014457 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0924 09:56:47.014471 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 09:56:47.014555 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:56:47.014815 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:56:47.014818 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:56:47.015087 (+ 269us) client_negotiation.cc:189] Negotiated authn=SASL
0924 09:56:47.015213 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:47.015220 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:47.016250 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:47.016256 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:47.017333 (+ 1077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:47.017347 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:47.018102 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:47.018108 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:47.018272 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:47.018773 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 09:56:47.018794 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 09:56:47.021402 (+ 2608us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:34237: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":116}
W20260924 09:56:47.021816 9368 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 09:56:47.014318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:59877 (local address 127.0.97.62:34237)
0924 09:56:47.014456 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0924 09:56:47.014460 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 09:56:47.014531 (+ 71us) server_negotiation.cc:407] Connection header received
0924 09:56:47.014628 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:56:47.014633 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:56:47.014709 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:56:47.014807 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:56:47.015355 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:47.016133 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:47.017450 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:47.017971 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:47.018068 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:47.021589 (+ 3521us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:59877: BlockingRecv error: recv got EOF from 127.0.97.3:59877 (error 108)
Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":42}
Sep 24 09:56:47 dist-test-slave-w8d2 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 09:56:47 dist-test-slave-w8d2 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 09:56:47.029220 9631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:47.022304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34237 (local address 127.0.97.3:48069)
0924 09:56:47.022863 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0924 09:56:47.022877 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 09:56:47.022977 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:56:47.023211 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:56:47.023214 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:56:47.023490 (+ 276us) client_negotiation.cc:189] Negotiated authn=SASL
0924 09:56:47.023637 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:47.023644 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:47.024500 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:47.024503 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:47.025422 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:47.025432 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:47.026032 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:47.026037 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:47.026151 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:47.026584 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 09:56:47.026600 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 09:56:47.029019 (+ 2419us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:34237: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":473}
W20260924 09:56:47.029386 9368 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 09:56:47.022386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:48069 (local address 127.0.97.62:34237)
0924 09:56:47.022587 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0924 09:56:47.022592 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 09:56:47.022927 (+ 335us) server_negotiation.cc:407] Connection header received
0924 09:56:47.023061 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:56:47.023064 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:56:47.023114 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:56:47.023189 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:56:47.023731 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:47.024408 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:47.025531 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:47.025932 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:47.026015 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:47.029177 (+ 3162us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:48069: BlockingRecv error: recv got EOF from 127.0.97.3:48069 (error 108)
Metrics: {"server-negotiator.queue_time_us":94}
W20260924 09:56:47.029481 9629 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 24 09:56:47 dist-test-slave-w8d2 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 09:56:47 dist-test-slave-w8d2 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 09:56:47.195268 9634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:47.187976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34237 (local address 127.0.97.1:37581)
0924 09:56:47.188400 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0924 09:56:47.188417 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 09:56:47.188535 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:56:47.188783 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:56:47.188786 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:56:47.189197 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0924 09:56:47.189381 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:47.189389 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:47.190249 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:47.190252 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:47.191152 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:47.191171 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:47.191828 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:47.191832 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:47.191941 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:47.192472 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 09:56:47.192489 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 09:56:47.195018 (+ 2529us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:34237: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":223,"threads_started":1}
W20260924 09:56:47.195451 9368 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 09:56:47.188032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:37581 (local address 127.0.97.62:34237)
0924 09:56:47.188230 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0924 09:56:47.188235 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 09:56:47.188488 (+ 253us) server_negotiation.cc:407] Connection header received
0924 09:56:47.188620 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:56:47.188624 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:56:47.188682 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:56:47.188758 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:56:47.189479 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:47.190151 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:47.191284 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:47.191702 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:47.191789 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:47.195215 (+ 3426us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:37581: BlockingRecv error: recv got EOF from 127.0.97.1:37581 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 24 09:56:47 dist-test-slave-w8d2 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 09:56:47 dist-test-slave-w8d2 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 09:56:47.617277 9635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:47.609718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34237 (local address 127.0.97.2:46865)
0924 09:56:47.610142 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0924 09:56:47.610159 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 09:56:47.610267 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:56:47.610518 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:56:47.610521 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:56:47.610942 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0924 09:56:47.611143 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:47.611152 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:47.612149 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:47.612153 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:47.613090 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:47.613108 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:47.613726 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:47.613732 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:47.613888 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:47.614475 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 09:56:47.614492 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 09:56:47.617054 (+ 2562us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:34237: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":210,"threads_started":1}
W20260924 09:56:47.617431 9368 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 09:56:47.609786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:46865 (local address 127.0.97.62:34237)
0924 09:56:47.609998 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0924 09:56:47.610004 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 09:56:47.610230 (+ 226us) server_negotiation.cc:407] Connection header received
0924 09:56:47.610353 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:56:47.610356 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:56:47.610416 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:56:47.610489 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:56:47.611281 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:47.612031 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:47.613205 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:47.613582 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:47.613664 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:47.617229 (+ 3565us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:46865: BlockingRecv error: recv got EOF from 127.0.97.2:46865 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 24 09:56:48 dist-test-slave-w8d2 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 09:56:48 dist-test-slave-w8d2 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 09:56:48.038110 9636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:48.030068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34237 (local address 127.0.97.3:42105)
0924 09:56:48.030523 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0924 09:56:48.030538 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 09:56:48.030654 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:56:48.030909 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:56:48.030912 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:56:48.031380 (+ 468us) client_negotiation.cc:189] Negotiated authn=SASL
0924 09:56:48.031570 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:48.031578 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:48.032632 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:48.032638 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:48.033894 (+ 1256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:48.033909 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:48.034536 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:48.034540 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:48.034671 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:48.035223 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 09:56:48.035276 (+ 53us) client_negotiation.cc:656] calling sasl_client_start()
0924 09:56:48.037898 (+ 2622us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:34237: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":242,"threads_started":1}
W20260924 09:56:48.038360 9368 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 09:56:48.030138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:42105 (local address 127.0.97.62:34237)
0924 09:56:48.030293 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0924 09:56:48.030298 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 09:56:48.030609 (+ 311us) server_negotiation.cc:407] Connection header received
0924 09:56:48.030742 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:56:48.030746 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:56:48.030804 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:56:48.030883 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:56:48.031676 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:48.032476 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:48.034025 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:48.034401 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:48.034489 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:48.038139 (+ 3650us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:42105: BlockingRecv error: recv got EOF from 127.0.97.3:42105 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 24 09:56:48 dist-test-slave-w8d2 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 09:56:48 dist-test-slave-w8d2 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 09:56:48.203545 9637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:48.196096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34237 (local address 127.0.97.1:42975)
0924 09:56:48.196563 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0924 09:56:48.196578 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 09:56:48.196698 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:56:48.196985 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:56:48.196989 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:56:48.197361 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0924 09:56:48.197508 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:48.197517 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:48.198501 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:48.198507 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:48.199430 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:48.199441 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:48.200063 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:48.200069 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:48.200218 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:48.200744 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 09:56:48.200761 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 09:56:48.203321 (+ 2560us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:34237: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":247,"threads_started":1}
W20260924 09:56:48.203655 9368 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 09:56:48.196204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:42975 (local address 127.0.97.62:34237)
0924 09:56:48.196396 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0924 09:56:48.196403 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 09:56:48.196651 (+ 248us) server_negotiation.cc:407] Connection header received
0924 09:56:48.196806 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:56:48.196811 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:56:48.196870 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:56:48.196945 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:56:48.197644 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:48.198373 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:48.199549 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:48.199929 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:48.200035 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:48.203458 (+ 3423us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:42975: BlockingRecv error: recv got EOF from 127.0.97.1:42975 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 24 09:56:48 dist-test-slave-w8d2 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 09:56:48 dist-test-slave-w8d2 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 09:56:48.625802 9638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:48.618151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34237 (local address 127.0.97.2:45835)
0924 09:56:48.618668 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0924 09:56:48.618684 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 09:56:48.618791 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:56:48.619132 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:56:48.619137 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:56:48.619572 (+ 435us) client_negotiation.cc:189] Negotiated authn=SASL
0924 09:56:48.619727 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:48.619737 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:48.620694 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:48.620700 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:48.621589 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:48.621600 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:48.622247 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:48.622253 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:48.622400 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:48.622905 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 09:56:48.622922 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 09:56:48.625556 (+ 2634us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:34237: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":292,"threads_started":1}
W20260924 09:56:48.625952 9368 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 09:56:48.618212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:45835 (local address 127.0.97.62:34237)
0924 09:56:48.618381 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0924 09:56:48.618386 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 09:56:48.618752 (+ 366us) server_negotiation.cc:407] Connection header received
0924 09:56:48.618910 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:56:48.618914 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:56:48.618995 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:56:48.619078 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:56:48.619864 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:48.620567 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:48.621721 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:48.622114 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:48.622223 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:48.625730 (+ 3507us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:45835: BlockingRecv error: recv got EOF from 127.0.97.2:45835 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 24 09:56:49 dist-test-slave-w8d2 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 09:56:49 dist-test-slave-w8d2 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 09:56:49.049808 9639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:49.038904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34237 (local address 127.0.97.3:45227)
0924 09:56:49.039386 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0924 09:56:49.039402 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 09:56:49.039524 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:56:49.039820 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:56:49.039824 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:56:49.040208 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0924 09:56:49.040346 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:49.040354 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:49.041306 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:49.041312 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:49.042197 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:49.042208 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:49.042856 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:49.042862 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:49.043011 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:49.046625 (+ 3614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 09:56:49.046642 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 09:56:49.049504 (+ 2862us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:34237: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":235,"threads_started":1}
W20260924 09:56:49.049911 9368 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 09:56:49.039068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:45227 (local address 127.0.97.62:34237)
0924 09:56:49.039269 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0924 09:56:49.039275 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 09:56:49.039476 (+ 201us) server_negotiation.cc:407] Connection header received
0924 09:56:49.039629 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:56:49.039633 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:56:49.039702 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:56:49.039778 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:56:49.040485 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:49.041181 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:49.042323 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:49.042724 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:49.042824 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:49.049699 (+ 6875us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:45227: BlockingRecv error: recv got EOF from 127.0.97.3:45227 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 24 09:56:49 dist-test-slave-w8d2 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 09:56:49 dist-test-slave-w8d2 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 09:56:49.211684 9640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:49.204369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34237 (local address 127.0.97.1:49323)
0924 09:56:49.204849 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0924 09:56:49.204866 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 09:56:49.204972 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:56:49.205232 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:56:49.205235 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:56:49.205552 (+ 317us) client_negotiation.cc:189] Negotiated authn=SASL
0924 09:56:49.205740 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:49.205749 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:49.206684 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:49.206688 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:49.207623 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:49.207635 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:49.208230 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:49.208235 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:49.208370 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:49.208896 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 09:56:49.208913 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 09:56:49.211403 (+ 2490us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:34237: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":244,"threads_started":1}
W20260924 09:56:49.211872 9368 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 09:56:49.204427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:49323 (local address 127.0.97.62:34237)
0924 09:56:49.204564 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0924 09:56:49.204569 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 09:56:49.204928 (+ 359us) server_negotiation.cc:407] Connection header received
0924 09:56:49.205056 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:56:49.205060 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:56:49.205117 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:56:49.205191 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:56:49.205847 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:49.206540 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:49.207725 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:49.208090 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:49.208175 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:49.211616 (+ 3441us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:49323: BlockingRecv error: recv got EOF from 127.0.97.1:49323 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 24 09:56:49 dist-test-slave-w8d2 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 09:56:49 dist-test-slave-w8d2 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 09:56:49.634240 9641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:49.626794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34237 (local address 127.0.97.2:34851)
0924 09:56:49.627289 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0924 09:56:49.627305 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 09:56:49.627419 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:56:49.627673 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:56:49.627676 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:56:49.627992 (+ 316us) client_negotiation.cc:189] Negotiated authn=SASL
0924 09:56:49.628148 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:49.628156 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:49.629135 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:49.629138 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:49.629997 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:49.630007 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:49.630602 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:49.630606 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:49.630728 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:49.631258 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 09:56:49.631276 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 09:56:49.633941 (+ 2665us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:34237: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":245,"threads_started":1}
W20260924 09:56:49.634393 9368 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 09:56:49.626869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:34851 (local address 127.0.97.62:34237)
0924 09:56:49.627012 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0924 09:56:49.627016 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 09:56:49.627375 (+ 359us) server_negotiation.cc:407] Connection header received
0924 09:56:49.627506 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:56:49.627509 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:56:49.627566 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:56:49.627641 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:56:49.628267 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:49.629040 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:49.630095 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:49.630500 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:49.630608 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:49.634132 (+ 3524us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:34851: BlockingRecv error: recv got EOF from 127.0.97.2:34851 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 24 09:56:50 dist-test-slave-w8d2 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 09:56:50 dist-test-slave-w8d2 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 09:56:50.058256 9642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:50.050629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34237 (local address 127.0.97.3:41319)
0924 09:56:50.051069 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0924 09:56:50.051084 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 09:56:50.051210 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:56:50.051567 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:56:50.051572 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:56:50.051985 (+ 413us) client_negotiation.cc:189] Negotiated authn=SASL
0924 09:56:50.052155 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:50.052164 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:50.053268 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:50.053274 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:50.054217 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:50.054229 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:50.054849 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:50.054854 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:50.055026 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:50.055538 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 09:56:50.055554 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 09:56:50.057999 (+ 2445us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:34237: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":236,"threads_started":1}
W20260924 09:56:50.058382 9368 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 09:56:50.050726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:41319 (local address 127.0.97.62:34237)
0924 09:56:50.050909 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0924 09:56:50.050915 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 09:56:50.051162 (+ 247us) server_negotiation.cc:407] Connection header received
0924 09:56:50.051351 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:56:50.051355 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:56:50.051436 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:56:50.051516 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:56:50.052287 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:50.053142 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:50.054337 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:50.054719 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:50.054819 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:50.058206 (+ 3387us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:41319: BlockingRecv error: recv got EOF from 127.0.97.3:41319 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 24 09:56:50 dist-test-slave-w8d2 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 09:56:50 dist-test-slave-w8d2 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 09:56:50.219971 9643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:50.212565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34237 (local address 127.0.97.1:55027)
0924 09:56:50.212969 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0924 09:56:50.212985 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 09:56:50.213095 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:56:50.213329 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:56:50.213332 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:56:50.213665 (+ 333us) client_negotiation.cc:189] Negotiated authn=SASL
0924 09:56:50.213804 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:50.213813 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:50.214896 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:50.214902 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:50.216061 (+ 1159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:50.216074 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:50.216651 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:50.216655 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:50.216784 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:50.217295 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 09:56:50.217311 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 09:56:50.219729 (+ 2418us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:34237: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":209,"threads_started":1}
W20260924 09:56:50.220106 9368 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 09:56:50.212641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:55027 (local address 127.0.97.62:34237)
0924 09:56:50.212858 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0924 09:56:50.212864 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 09:56:50.213052 (+ 188us) server_negotiation.cc:407] Connection header received
0924 09:56:50.213180 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:56:50.213184 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:56:50.213243 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:56:50.213315 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:56:50.213908 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:50.214745 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:50.216179 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:50.216546 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:50.216635 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:50.219910 (+ 3275us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:55027: BlockingRecv error: recv got EOF from 127.0.97.1:55027 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
Sep 24 09:56:50 dist-test-slave-w8d2 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 09:56:50 dist-test-slave-w8d2 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 09:56:50.642761 9644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:50.635105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34237 (local address 127.0.97.2:44213)
0924 09:56:50.635563 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0924 09:56:50.635578 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 09:56:50.635687 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:56:50.635939 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:56:50.635941 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:56:50.636311 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0924 09:56:50.636477 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:50.636490 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:50.637730 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:50.637736 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:50.638669 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:50.638681 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:50.639331 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:50.639336 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:50.639463 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:50.640032 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 09:56:50.640049 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 09:56:50.642537 (+ 2488us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:34237: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":266,"threads_started":1}
W20260924 09:56:50.642889 9368 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 09:56:50.635182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:44213 (local address 127.0.97.62:34237)
0924 09:56:50.635390 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0924 09:56:50.635398 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 09:56:50.635652 (+ 254us) server_negotiation.cc:407] Connection header received
0924 09:56:50.635774 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:56:50.635777 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:56:50.635837 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:56:50.635912 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:56:50.636653 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:50.637605 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:50.638786 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:50.639192 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:50.639310 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:50.642706 (+ 3396us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:44213: BlockingRecv error: recv got EOF from 127.0.97.2:44213 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Sep 24 09:56:51 dist-test-slave-w8d2 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 09:56:51 dist-test-slave-w8d2 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 09:56:51.066308 9645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:51.059152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34237 (local address 127.0.97.3:47531)
0924 09:56:51.059605 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0924 09:56:51.059621 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 09:56:51.059725 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:56:51.059978 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:56:51.059981 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:56:51.060368 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0924 09:56:51.060511 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:51.060519 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:51.061437 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:51.061441 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:51.062367 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:51.062377 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:51.062981 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:51.062985 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:51.063131 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:51.063674 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 09:56:51.063690 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 09:56:51.066096 (+ 2406us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:34237: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":250,"threads_started":1}
W20260924 09:56:51.066457 9368 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 09:56:51.059202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:47531 (local address 127.0.97.62:34237)
0924 09:56:51.059374 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0924 09:56:51.059378 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 09:56:51.059680 (+ 302us) server_negotiation.cc:407] Connection header received
0924 09:56:51.059807 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:56:51.059810 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:56:51.059878 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:56:51.059953 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:56:51.060634 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:51.061344 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:51.062478 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:51.062875 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:51.062958 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:51.066258 (+ 3300us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:47531: BlockingRecv error: recv got EOF from 127.0.97.3:47531 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 24 09:56:51 dist-test-slave-w8d2 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 09:56:51 dist-test-slave-w8d2 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 09:56:51.228053 9646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:51.220760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34237 (local address 127.0.97.1:36489)
0924 09:56:51.221229 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0924 09:56:51.221245 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 09:56:51.221350 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:56:51.221591 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:56:51.221594 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:56:51.221957 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0924 09:56:51.222141 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:51.222150 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:51.223068 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:51.223072 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:51.223986 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:51.223998 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:51.224569 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:51.224572 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:51.224683 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:51.225253 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 09:56:51.225269 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 09:56:51.227823 (+ 2554us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:34237: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":222,"threads_started":1}
W20260924 09:56:51.228163 9368 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 09:56:51.220832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:36489 (local address 127.0.97.62:34237)
0924 09:56:51.220997 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0924 09:56:51.221001 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 09:56:51.221307 (+ 306us) server_negotiation.cc:407] Connection header received
0924 09:56:51.221431 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:56:51.221434 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:56:51.221490 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:56:51.221563 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:56:51.222251 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:51.222932 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:51.224087 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:51.224470 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:51.224552 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:51.227994 (+ 3442us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:36489: BlockingRecv error: recv got EOF from 127.0.97.1:36489 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 24 09:56:51 dist-test-slave-w8d2 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 09:56:51 dist-test-slave-w8d2 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 09:56:51.651309 9647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:51.643632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34237 (local address 127.0.97.2:44897)
0924 09:56:51.644080 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0924 09:56:51.644094 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 09:56:51.644209 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:56:51.644490 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:56:51.644495 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:56:51.644908 (+ 413us) client_negotiation.cc:189] Negotiated authn=SASL
0924 09:56:51.645059 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:51.645067 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:51.646016 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:51.646021 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:51.647012 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:51.647023 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:51.647783 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:51.647789 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:51.647947 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:51.648476 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 09:56:51.648492 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 09:56:51.650967 (+ 2475us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:34237: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":198,"threads_started":1}
W20260924 09:56:51.651423 9368 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 09:56:51.643806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:44897 (local address 127.0.97.62:34237)
0924 09:56:51.644004 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0924 09:56:51.644011 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 09:56:51.644158 (+ 147us) server_negotiation.cc:407] Connection header received
0924 09:56:51.644312 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:56:51.644317 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:56:51.644380 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:56:51.644453 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:56:51.645191 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:51.645893 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:51.647136 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:51.647646 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:51.647752 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:51.651154 (+ 3402us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:44897: BlockingRecv error: recv got EOF from 127.0.97.2:44897 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 24 09:56:52 dist-test-slave-w8d2 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 09:56:52 dist-test-slave-w8d2 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 09:56:52.074658 9648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:52.067096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34237 (local address 127.0.97.3:35317)
0924 09:56:52.067586 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0924 09:56:52.067602 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 09:56:52.067721 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:56:52.068030 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:56:52.068034 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:56:52.068437 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0924 09:56:52.068609 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:52.068618 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:52.069595 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:52.069601 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:52.070500 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:52.070520 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:52.071166 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:52.071172 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:52.071364 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:52.071868 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 09:56:52.071883 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 09:56:52.074327 (+ 2444us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:34237: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":246,"threads_started":1}
W20260924 09:56:52.074775 9368 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 09:56:52.067169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:35317 (local address 127.0.97.62:34237)
0924 09:56:52.067364 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0924 09:56:52.067370 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 09:56:52.067679 (+ 309us) server_negotiation.cc:407] Connection header received
0924 09:56:52.067829 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:56:52.067833 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:56:52.067906 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:56:52.067983 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:56:52.068744 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:52.069468 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:52.070640 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:52.071039 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:52.071141 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:52.074526 (+ 3385us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:35317: BlockingRecv error: recv got EOF from 127.0.97.3:35317 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 24 09:56:52 dist-test-slave-w8d2 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 09:56:52 dist-test-slave-w8d2 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 09:56:52.236691 9649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:52.229013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34237 (local address 127.0.97.1:39047)
0924 09:56:52.229577 (+ 564us) negotiation.cc:107] Waiting for socket to connect
0924 09:56:52.229597 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 09:56:52.229733 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:56:52.230034 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:56:52.230039 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:56:52.230436 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0924 09:56:52.230573 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:52.230582 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:52.231604 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:52.231610 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:52.232504 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:52.232515 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:52.233204 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:52.233209 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:52.233368 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:52.233923 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 09:56:52.233939 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 09:56:52.236482 (+ 2543us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:34237: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":294,"threads_started":1}
W20260924 09:56:52.236886 9368 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 09:56:52.229072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:39047 (local address 127.0.97.62:34237)
0924 09:56:52.229246 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0924 09:56:52.229253 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 09:56:52.229665 (+ 412us) server_negotiation.cc:407] Connection header received
0924 09:56:52.229841 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:56:52.229846 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:56:52.229911 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:56:52.229986 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:56:52.230706 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:52.231482 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:52.232643 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:52.233062 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:52.233165 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:52.236673 (+ 3508us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:39047: BlockingRecv error: recv got EOF from 127.0.97.1:39047 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 24 09:56:52 dist-test-slave-w8d2 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 09:56:52 dist-test-slave-w8d2 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 09:56:52.659305 9650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:52.652156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34237 (local address 127.0.97.2:33329)
0924 09:56:52.652641 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0924 09:56:52.652656 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 09:56:52.652781 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:56:52.653023 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:56:52.653026 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:56:52.653400 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0924 09:56:52.653536 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:52.653544 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:52.654430 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:52.654434 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:52.655371 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:52.655382 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:52.655983 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:52.655987 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:52.656100 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:52.656598 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 09:56:52.656614 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 09:56:52.659011 (+ 2397us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:34237: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":246,"threads_started":1}
W20260924 09:56:52.659437 9368 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 09:56:52.652188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:33329 (local address 127.0.97.62:34237)
0924 09:56:52.652328 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0924 09:56:52.652333 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 09:56:52.652732 (+ 399us) server_negotiation.cc:407] Connection header received
0924 09:56:52.652864 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:56:52.652867 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:56:52.652925 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:56:52.652996 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:56:52.653654 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:52.654336 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:52.655479 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:52.655873 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:52.655961 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:52.659196 (+ 3235us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:33329: BlockingRecv error: recv got EOF from 127.0.97.2:33329 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Sep 24 09:56:53 dist-test-slave-w8d2 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 09:56:53 dist-test-slave-w8d2 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 09:56:53.082647 9651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:53.075568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34237 (local address 127.0.97.3:36309)
0924 09:56:53.075987 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0924 09:56:53.076002 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 09:56:53.076107 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:56:53.076376 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:56:53.076379 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:56:53.076721 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0924 09:56:53.076866 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:53.076873 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:53.077764 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:53.077767 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:53.078650 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:53.078660 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:53.079329 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:53.079335 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:53.079461 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:53.079991 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 09:56:53.080007 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 09:56:53.082438 (+ 2431us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:34237: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":231,"threads_started":1}
W20260924 09:56:53.082813 9368 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 09:56:53.075636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:36309 (local address 127.0.97.62:34237)
0924 09:56:53.075856 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0924 09:56:53.075862 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 09:56:53.076070 (+ 208us) server_negotiation.cc:407] Connection header received
0924 09:56:53.076193 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:56:53.076196 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:56:53.076270 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:56:53.076347 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:56:53.076966 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:53.077667 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:53.078754 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:53.079164 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:53.079280 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:53.082611 (+ 3331us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:36309: BlockingRecv error: recv got EOF from 127.0.97.3:36309 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Sep 24 09:56:53 dist-test-slave-w8d2 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 09:56:53 dist-test-slave-w8d2 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 09:56:53.249418 9652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:53.237587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34237 (local address 127.0.97.1:44443)
0924 09:56:53.238133 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0924 09:56:53.238149 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 09:56:53.238281 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:56:53.238600 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:56:53.238605 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:56:53.239003 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0924 09:56:53.239172 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:53.239181 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:53.240158 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:53.240162 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:53.241057 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:53.241069 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:53.241741 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:53.241747 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:53.241899 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:53.246004 (+ 4105us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 09:56:53.246022 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 09:56:53.249084 (+ 3062us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:34237: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":240,"threads_started":1}
W20260924 09:56:53.249514 9368 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 09:56:53.237654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:44443 (local address 127.0.97.62:34237)
0924 09:56:53.237822 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0924 09:56:53.237828 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 09:56:53.238276 (+ 448us) server_negotiation.cc:407] Connection header received
0924 09:56:53.238397 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:56:53.238402 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:56:53.238461 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:56:53.238538 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:56:53.239316 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:53.240023 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:53.241189 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:53.241598 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:53.241702 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:53.249309 (+ 7607us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:44443: BlockingRecv error: recv got EOF from 127.0.97.1:44443 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 24 09:56:53 dist-test-slave-w8d2 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 09:56:53 dist-test-slave-w8d2 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 09:56:53.667796 9653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:53.660141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34237 (local address 127.0.97.2:49965)
0924 09:56:53.660735 (+ 594us) negotiation.cc:107] Waiting for socket to connect
0924 09:56:53.660750 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 09:56:53.660872 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:56:53.661173 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:56:53.661178 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:56:53.661579 (+ 401us) client_negotiation.cc:189] Negotiated authn=SASL
0924 09:56:53.661745 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:53.661754 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:53.662716 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:53.662721 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:53.663640 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:53.663651 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:53.664283 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:53.664288 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:53.664441 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:53.664969 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 09:56:53.664985 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 09:56:53.667473 (+ 2488us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:34237: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":256,"threads_started":1}
W20260924 09:56:53.667874 9368 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 09:56:53.660188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:49965 (local address 127.0.97.62:34237)
0924 09:56:53.660376 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0924 09:56:53.660382 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 09:56:53.660830 (+ 448us) server_negotiation.cc:407] Connection header received
0924 09:56:53.660983 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:56:53.660988 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:56:53.661056 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:56:53.661129 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:56:53.661878 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:53.662589 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:53.663766 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:53.664153 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:53.664255 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:53.667659 (+ 3404us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:49965: BlockingRecv error: recv got EOF from 127.0.97.2:49965 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 24 09:56:54 dist-test-slave-w8d2 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 09:56:54 dist-test-slave-w8d2 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 09:56:54.096709 9654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:54.083475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34237 (local address 127.0.97.3:41483)
0924 09:56:54.084042 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0924 09:56:54.084057 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 09:56:54.084169 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:56:54.084455 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:56:54.084459 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:56:54.084818 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0924 09:56:54.084986 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:54.084994 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:54.085900 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:54.085903 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:54.086818 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:54.086831 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:54.087527 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:54.087531 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:54.087637 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:54.093653 (+ 6016us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 09:56:54.093671 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 09:56:54.096381 (+ 2710us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:34237: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":242,"threads_started":1}
W20260924 09:56:54.096854 9368 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 09:56:54.083585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:41483 (local address 127.0.97.62:34237)
0924 09:56:54.083741 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0924 09:56:54.083745 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 09:56:54.084121 (+ 376us) server_negotiation.cc:407] Connection header received
0924 09:56:54.084254 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:56:54.084257 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:56:54.084328 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:56:54.084417 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:56:54.085088 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:54.085798 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:54.086941 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:54.087410 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:54.087500 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:54.096622 (+ 9122us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:41483: BlockingRecv error: recv got EOF from 127.0.97.3:41483 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 24 09:56:54 dist-test-slave-w8d2 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 09:56:54 dist-test-slave-w8d2 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 09:56:54.258172 9655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:54.250291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34237 (local address 127.0.97.1:47141)
0924 09:56:54.250791 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0924 09:56:54.250807 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 09:56:54.250927 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:56:54.251198 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:56:54.251201 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:56:54.251643 (+ 442us) client_negotiation.cc:189] Negotiated authn=SASL
0924 09:56:54.251811 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:54.251822 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:54.253019 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:54.253023 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:54.254112 (+ 1089us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:54.254124 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:54.254777 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:54.254781 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:54.254902 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:54.255478 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 09:56:54.255495 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 09:56:54.257935 (+ 2440us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:34237: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":273,"threads_started":1}
W20260924 09:56:54.258378 9368 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 09:56:54.250442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:47141 (local address 127.0.97.62:34237)
0924 09:56:54.250593 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0924 09:56:54.250597 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 09:56:54.250873 (+ 276us) server_negotiation.cc:407] Connection header received
0924 09:56:54.251017 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:56:54.251020 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:56:54.251080 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:56:54.251160 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:56:54.251962 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:54.252893 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:54.254224 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:54.254648 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:54.254735 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:54.258138 (+ 3403us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:47141: BlockingRecv error: recv got EOF from 127.0.97.1:47141 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 24 09:56:54 dist-test-slave-w8d2 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 09:56:54 dist-test-slave-w8d2 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 09:56:54.676386 9656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:54.668710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34237 (local address 127.0.97.2:47765)
0924 09:56:54.669194 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0924 09:56:54.669213 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 09:56:54.669361 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:56:54.669668 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:56:54.669672 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:56:54.670114 (+ 442us) client_negotiation.cc:189] Negotiated authn=SASL
0924 09:56:54.670258 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:54.670266 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:54.671370 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:54.671375 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:54.672264 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:54.672276 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:54.672954 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:54.672960 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:54.673108 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:54.673648 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 09:56:54.673665 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 09:56:54.676169 (+ 2504us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:34237: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":228,"threads_started":1}
W20260924 09:56:54.676561 9368 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 09:56:54.668778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:47765 (local address 127.0.97.62:34237)
0924 09:56:54.668945 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0924 09:56:54.668951 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 09:56:54.669295 (+ 344us) server_negotiation.cc:407] Connection header received
0924 09:56:54.669476 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:56:54.669481 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:56:54.669544 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:56:54.669623 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:56:54.670387 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:54.671223 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:54.672387 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:54.672799 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:54.672899 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:54.676380 (+ 3481us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:47765: BlockingRecv error: recv got EOF from 127.0.97.2:47765 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 24 09:56:55 dist-test-slave-w8d2 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 09:56:55 dist-test-slave-w8d2 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 09:56:55.105134 9657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:55.097615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34237 (local address 127.0.97.3:37641)
0924 09:56:55.098099 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0924 09:56:55.098115 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 09:56:55.098221 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:56:55.098513 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:56:55.098516 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:56:55.098887 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0924 09:56:55.099033 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:55.099041 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:55.100114 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:55.100117 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:55.101004 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:55.101015 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:55.101657 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:55.101661 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:55.101772 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:55.102334 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 09:56:55.102352 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 09:56:55.104913 (+ 2561us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:34237: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":266,"threads_started":1}
W20260924 09:56:55.105286 9368 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 09:56:55.097700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:37641 (local address 127.0.97.62:34237)
0924 09:56:55.097873 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0924 09:56:55.097878 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 09:56:55.098177 (+ 299us) server_negotiation.cc:407] Connection header received
0924 09:56:55.098307 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:56:55.098310 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:56:55.098405 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:56:55.098486 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:56:55.099182 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:55.099997 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:55.101114 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:55.101527 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:55.101613 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:55.105095 (+ 3482us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:37641: BlockingRecv error: recv got EOF from 127.0.97.3:37641 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 24 09:56:55 dist-test-slave-w8d2 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 09:56:55 dist-test-slave-w8d2 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 09:56:55.267913 9658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:55.259465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34237 (local address 127.0.97.1:39459)
0924 09:56:55.260056 (+ 591us) negotiation.cc:107] Waiting for socket to connect
0924 09:56:55.260072 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 09:56:55.260201 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:56:55.260501 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:56:55.260506 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:56:55.260887 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0924 09:56:55.261053 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:55.261062 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:55.262335 (+ 1273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:55.262341 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:55.263633 (+ 1292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:55.263646 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:55.264410 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:55.264417 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:55.264567 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:55.265058 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 09:56:55.265074 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 09:56:55.267688 (+ 2614us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:34237: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":358,"threads_started":1}
W20260924 09:56:55.268057 9368 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 09:56:55.259465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:39459 (local address 127.0.97.62:34237)
0924 09:56:55.259664 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0924 09:56:55.259671 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 09:56:55.260157 (+ 486us) server_negotiation.cc:407] Connection header received
0924 09:56:55.260314 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:56:55.260320 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:56:55.260380 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:56:55.260457 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:56:55.261195 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:55.262209 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:55.263762 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:55.264244 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:55.264405 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:55.267866 (+ 3461us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:39459: BlockingRecv error: recv got EOF from 127.0.97.1:39459 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Sep 24 09:56:55 dist-test-slave-w8d2 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 09:56:55 dist-test-slave-w8d2 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 09:56:55.684456 9659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:55.677181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34237 (local address 127.0.97.2:36451)
0924 09:56:55.677611 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0924 09:56:55.677633 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 09:56:55.677737 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:56:55.677983 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:56:55.677986 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:56:55.678328 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0924 09:56:55.678490 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:55.678498 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:55.679442 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:55.679446 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:55.680344 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:55.680355 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:55.680939 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:55.680943 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:55.681065 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:55.681545 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 09:56:55.681562 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 09:56:55.684212 (+ 2650us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:34237: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":222,"threads_started":1}
W20260924 09:56:55.684584 9368 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 09:56:55.677267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:36451 (local address 127.0.97.62:34237)
0924 09:56:55.677457 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0924 09:56:55.677464 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 09:56:55.677700 (+ 236us) server_negotiation.cc:407] Connection header received
0924 09:56:55.677821 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:56:55.677824 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:56:55.677884 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:56:55.677958 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:56:55.678592 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:55.679345 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:55.680446 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:55.680834 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:55.680917 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:55.684383 (+ 3466us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:36451: BlockingRecv error: recv got EOF from 127.0.97.2:36451 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 24 09:56:56 dist-test-slave-w8d2 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 09:56:56 dist-test-slave-w8d2 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 09:56:56.114223 9660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:56.106009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34237 (local address 127.0.97.3:37685)
0924 09:56:56.106517 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0924 09:56:56.106532 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 09:56:56.106640 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:56:56.106901 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:56:56.106904 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:56:56.107298 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0924 09:56:56.107460 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:56.107468 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:56.108427 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:56.108430 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:56.109412 (+ 982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:56.109426 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:56.110205 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:56.110211 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:56.110372 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:56.111011 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 09:56:56.111035 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 09:56:56.113900 (+ 2865us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:34237: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":284,"threads_started":1}
W20260924 09:56:56.114327 9368 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 09:56:56.106099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:37685 (local address 127.0.97.62:34237)
0924 09:56:56.106250 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0924 09:56:56.106255 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 09:56:56.106596 (+ 341us) server_negotiation.cc:407] Connection header received
0924 09:56:56.106735 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:56:56.106739 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:56:56.106799 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:56:56.106869 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:56:56.107622 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:56.108326 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:56.109578 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:56.110059 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:56.110179 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:56.114123 (+ 3944us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:37685: BlockingRecv error: recv got EOF from 127.0.97.3:37685 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 24 09:56:56 dist-test-slave-w8d2 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 09:56:56 dist-test-slave-w8d2 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 09:56:56.276701 9661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:56.268774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34237 (local address 127.0.97.1:55901)
0924 09:56:56.269234 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0924 09:56:56.269249 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 09:56:56.269368 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:56:56.269678 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:56:56.269682 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:56:56.270043 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0924 09:56:56.270186 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:56.270204 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:56.271164 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:56.271169 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:56.272117 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:56.272128 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:56.272832 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:56.272838 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:56.273005 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:56.273541 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 09:56:56.273556 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 09:56:56.276483 (+ 2927us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:34237: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":254,"threads_started":1}
W20260924 09:56:56.276860 9368 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 09:56:56.268854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:55901 (local address 127.0.97.62:34237)
0924 09:56:56.269022 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0924 09:56:56.269029 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 09:56:56.269323 (+ 294us) server_negotiation.cc:407] Connection header received
0924 09:56:56.269477 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:56:56.269482 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:56:56.269555 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:56:56.269635 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:56:56.270325 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:56.271042 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:56.272246 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:56.272693 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:56.272804 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:56.276674 (+ 3870us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:55901: BlockingRecv error: recv got EOF from 127.0.97.1:55901 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 24 09:56:56 dist-test-slave-w8d2 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 09:56:56 dist-test-slave-w8d2 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 09:56:56.694060 9662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:56.685301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34237 (local address 127.0.97.2:49305)
0924 09:56:56.685764 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0924 09:56:56.685780 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 09:56:56.685882 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:56:56.686127 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:56:56.686130 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:56:56.686466 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
0924 09:56:56.686617 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:56.686629 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:56.687603 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:56.687607 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:56.688985 (+ 1378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:56.689000 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:56.689822 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:56.689828 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:56.689998 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:56.690660 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 09:56:56.690681 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 09:56:56.693764 (+ 3083us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:34237: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":258,"threads_started":1}
W20260924 09:56:56.694164 9368 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 09:56:56.685354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:49305 (local address 127.0.97.62:34237)
0924 09:56:56.685529 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0924 09:56:56.685535 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 09:56:56.685879 (+ 344us) server_negotiation.cc:407] Connection header received
0924 09:56:56.685967 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:56:56.685971 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:56:56.686032 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:56:56.686118 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:56:56.686760 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:56.687503 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:56.689131 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:56.689670 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:56.689780 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:56.693931 (+ 4151us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:49305: BlockingRecv error: recv got EOF from 127.0.97.2:49305 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 24 09:56:57 dist-test-slave-w8d2 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 09:56:57 dist-test-slave-w8d2 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 09:56:57.122400 9663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:57.115150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34237 (local address 127.0.97.3:42115)
0924 09:56:57.115762 (+ 612us) negotiation.cc:107] Waiting for socket to connect
0924 09:56:57.115777 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 09:56:57.115886 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:56:57.116145 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:56:57.116148 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:56:57.116475 (+ 327us) client_negotiation.cc:189] Negotiated authn=SASL
0924 09:56:57.116610 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:57.116618 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:57.117510 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:57.117514 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:57.118415 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:57.118426 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:57.119055 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:57.119059 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:57.119176 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:57.119723 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 09:56:57.119739 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 09:56:57.122190 (+ 2451us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:34237: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":512,"thread_start_us":408,"threads_started":1}
W20260924 09:56:57.122596 9368 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 09:56:57.115329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:42115 (local address 127.0.97.62:34237)
0924 09:56:57.115531 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0924 09:56:57.115535 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 09:56:57.115853 (+ 318us) server_negotiation.cc:407] Connection header received
0924 09:56:57.115970 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:56:57.115974 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:56:57.116035 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:56:57.116111 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:56:57.116713 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:57.117409 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:57.118531 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:57.118946 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:57.119037 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:57.122372 (+ 3335us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:42115: BlockingRecv error: recv got EOF from 127.0.97.3:42115 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
Sep 24 09:56:57 dist-test-slave-w8d2 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 09:56:57 dist-test-slave-w8d2 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 09:56:57.285354 9664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:57.277565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34237 (local address 127.0.97.1:52329)
0924 09:56:57.278029 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0924 09:56:57.278044 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 09:56:57.278171 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:56:57.278499 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:56:57.278504 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:56:57.278909 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0924 09:56:57.279072 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:57.279080 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:57.280088 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:57.280093 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:57.281130 (+ 1037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:57.281144 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:57.281896 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:57.281901 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:57.282072 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:57.282656 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 09:56:57.282672 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 09:56:57.285129 (+ 2457us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:34237: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":241,"threads_started":1}
W20260924 09:56:57.285490 9368 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 09:56:57.277717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:52329 (local address 127.0.97.62:34237)
0924 09:56:57.277928 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0924 09:56:57.277934 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 09:56:57.278122 (+ 188us) server_negotiation.cc:407] Connection header received
0924 09:56:57.278287 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:56:57.278292 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:56:57.278373 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:56:57.278453 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:56:57.279228 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:57.279965 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:57.281273 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:57.281742 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:57.281851 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:57.285318 (+ 3467us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:52329: BlockingRecv error: recv got EOF from 127.0.97.1:52329 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Sep 24 09:56:57 dist-test-slave-w8d2 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 09:56:57 dist-test-slave-w8d2 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 09:56:57.702929 9665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:57.694916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34237 (local address 127.0.97.2:46713)
0924 09:56:57.695400 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0924 09:56:57.695419 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 09:56:57.695546 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:56:57.695937 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:56:57.695942 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:56:57.696434 (+ 492us) client_negotiation.cc:189] Negotiated authn=SASL
0924 09:56:57.696643 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:57.696653 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:57.697842 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:57.697846 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:57.698720 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:57.698732 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:57.699374 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:57.699379 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:57.699493 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:57.700020 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 09:56:57.700036 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 09:56:57.702687 (+ 2651us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:34237: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":228,"threads_started":1}
W20260924 09:56:57.703071 9368 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 09:56:57.694976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:46713 (local address 127.0.97.62:34237)
0924 09:56:57.695142 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0924 09:56:57.695148 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 09:56:57.695500 (+ 352us) server_negotiation.cc:407] Connection header received
0924 09:56:57.695701 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:56:57.695706 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:56:57.695784 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:56:57.695892 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:56:57.696860 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:57.697717 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:57.698831 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:57.699224 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:57.699341 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:57.702888 (+ 3547us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:46713: BlockingRecv error: recv got EOF from 127.0.97.2:46713 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 24 09:56:58 dist-test-slave-w8d2 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 09:56:58 dist-test-slave-w8d2 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 09:56:58.130667 9666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:58.123125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34237 (local address 127.0.97.3:45279)
0924 09:56:58.123588 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0924 09:56:58.123604 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 09:56:58.123714 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:56:58.123968 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:56:58.123971 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:56:58.124376 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0924 09:56:58.124554 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:58.124563 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:58.125515 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:58.125518 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:58.126444 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:58.126455 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:58.127025 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:58.127029 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:58.127136 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:58.127666 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 09:56:58.127682 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 09:56:58.130448 (+ 2766us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:34237: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":271,"threads_started":1}
W20260924 09:56:58.130791 9368 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 09:56:58.123204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:45279 (local address 127.0.97.62:34237)
0924 09:56:58.123385 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0924 09:56:58.123389 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 09:56:58.123668 (+ 279us) server_negotiation.cc:407] Connection header received
0924 09:56:58.123801 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:56:58.123804 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:56:58.123861 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:56:58.123936 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:56:58.124653 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:58.125408 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:58.126555 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:58.126922 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:58.127007 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:58.130601 (+ 3594us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:45279: BlockingRecv error: recv got EOF from 127.0.97.3:45279 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 24 09:56:58 dist-test-slave-w8d2 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 09:56:58 dist-test-slave-w8d2 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 09:56:58.294106 9667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:58.286128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34237 (local address 127.0.97.1:50439)
0924 09:56:58.286611 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0924 09:56:58.286627 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 09:56:58.286736 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:56:58.287037 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:56:58.287041 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:56:58.287454 (+ 413us) client_negotiation.cc:189] Negotiated authn=SASL
0924 09:56:58.287613 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:58.287622 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:58.288618 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:58.288623 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:58.289555 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:58.289567 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:58.290265 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:58.290269 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:58.290400 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:58.290983 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 09:56:58.290999 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 09:56:58.293824 (+ 2825us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:34237: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":261,"threads_started":1}
W20260924 09:56:58.294272 9368 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 09:56:58.286188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:50439 (local address 127.0.97.62:34237)
0924 09:56:58.286357 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0924 09:56:58.286362 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 09:56:58.286687 (+ 325us) server_negotiation.cc:407] Connection header received
0924 09:56:58.286816 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:56:58.286819 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:56:58.286875 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:56:58.286960 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:56:58.287727 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:58.288489 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:58.289664 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:58.290127 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:58.290211 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:58.294040 (+ 3829us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:50439: BlockingRecv error: recv got EOF from 127.0.97.1:50439 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 24 09:56:58 dist-test-slave-w8d2 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 09:56:58 dist-test-slave-w8d2 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 09:56:58.711079 9668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:58.703757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34237 (local address 127.0.97.2:36683)
0924 09:56:58.704254 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0924 09:56:58.704270 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 09:56:58.704382 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:56:58.704618 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:56:58.704622 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:56:58.704996 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0924 09:56:58.705146 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:58.705154 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:58.706031 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:58.706034 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:58.706930 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:58.706941 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:58.707541 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:58.707545 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:58.707664 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:58.708238 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 09:56:58.708255 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 09:56:58.710837 (+ 2582us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:34237: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":256,"threads_started":1}
W20260924 09:56:58.711195 9368 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 09:56:58.703816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:36683 (local address 127.0.97.62:34237)
0924 09:56:58.703970 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0924 09:56:58.703984 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 09:56:58.704331 (+ 347us) server_negotiation.cc:407] Connection header received
0924 09:56:58.704460 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:56:58.704463 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:56:58.704518 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:56:58.704591 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:56:58.705249 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:58.705907 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:58.707029 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:58.707447 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:58.707527 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:58.711027 (+ 3500us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:36683: BlockingRecv error: recv got EOF from 127.0.97.2:36683 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 24 09:56:59 dist-test-slave-w8d2 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 09:56:59 dist-test-slave-w8d2 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 09:56:59.142825 9669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:59.131495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34237 (local address 127.0.97.3:55443)
0924 09:56:59.131958 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0924 09:56:59.131974 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 09:56:59.132114 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:56:59.132416 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:56:59.132421 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:56:59.132783 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0924 09:56:59.132930 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:59.132939 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:59.133987 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:59.133991 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:59.135042 (+ 1051us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:59.135058 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:59.135784 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:59.135789 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:59.135951 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:59.140031 (+ 4080us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 09:56:59.140048 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 09:56:59.142525 (+ 2477us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:34237: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":249,"threads_started":1}
W20260924 09:56:59.142975 9368 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 09:56:59.131528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:55443 (local address 127.0.97.62:34237)
0924 09:56:59.131705 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0924 09:56:59.131711 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 09:56:59.132057 (+ 346us) server_negotiation.cc:407] Connection header received
0924 09:56:59.132220 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:56:59.132225 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:56:59.132282 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:56:59.132359 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:56:59.133068 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:59.133855 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:59.135162 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:59.135661 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:59.135762 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:59.142759 (+ 6997us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:55443: BlockingRecv error: recv got EOF from 127.0.97.3:55443 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 24 09:56:59 dist-test-slave-w8d2 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 09:56:59 dist-test-slave-w8d2 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 09:56:59.302579 9670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:59.294912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34237 (local address 127.0.97.1:59853)
0924 09:56:59.295373 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0924 09:56:59.295388 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 09:56:59.295500 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:56:59.295754 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:56:59.295757 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:56:59.296149 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0924 09:56:59.296298 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:59.296307 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:59.297262 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:59.297266 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:59.298290 (+ 1024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:59.298303 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:59.299185 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:59.299191 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:59.299378 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:59.299909 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 09:56:59.299925 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 09:56:59.302355 (+ 2430us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:34237: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":207,"threads_started":1}
W20260924 09:56:59.302704 9368 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 09:56:59.294995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:59853 (local address 127.0.97.62:34237)
0924 09:56:59.295215 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0924 09:56:59.295221 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 09:56:59.295455 (+ 234us) server_negotiation.cc:407] Connection header received
0924 09:56:59.295593 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:56:59.295596 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:56:59.295656 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:56:59.295729 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:56:59.296415 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:59.297115 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:59.298410 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:59.298993 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:59.299114 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:59.302514 (+ 3400us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:59853: BlockingRecv error: recv got EOF from 127.0.97.1:59853 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Sep 24 09:56:59 dist-test-slave-w8d2 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 09:56:59 dist-test-slave-w8d2 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260924 09:56:59.719748 9368 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 09:56:59.712101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:54087 (local address 127.0.97.62:34237)
0924 09:56:59.712300 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0924 09:56:59.712305 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 09:56:59.712479 (+ 174us) server_negotiation.cc:407] Connection header received
0924 09:56:59.712651 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:56:59.712656 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:56:59.712718 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:56:59.712794 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:56:59.713522 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:59.714260 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:59.715473 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:56:59.715908 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:56:59.716017 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:59.719506 (+ 3489us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:54087: BlockingRecv error: recv got EOF from 127.0.97.2:54087 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
I20260924 09:56:59.720206 9671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:56:59.711924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34237 (local address 127.0.97.2:54087)
0924 09:56:59.712389 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0924 09:56:59.712404 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 09:56:59.712539 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:56:59.712838 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:56:59.712842 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:56:59.713231 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0924 09:56:59.713383 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:59.713391 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:59.714384 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:59.714389 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:59.715344 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:56:59.715356 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:56:59.716080 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:56:59.716086 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:56:59.716231 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:56:59.716720 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 09:56:59.716737 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 09:56:59.719960 (+ 3223us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:34237: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":220,"threads_started":1}
Sep 24 09:57:00 dist-test-slave-w8d2 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 09:57:00 dist-test-slave-w8d2 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 09:57:00.151820 9672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:57:00.143772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34237 (local address 127.0.97.3:34445)
0924 09:57:00.144257 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0924 09:57:00.144277 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 09:57:00.144410 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:57:00.144708 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:57:00.144712 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:57:00.145108 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0924 09:57:00.145327 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:57:00.145340 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:57:00.146232 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:57:00.146237 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:57:00.147257 (+ 1020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:57:00.147272 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:57:00.147963 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:57:00.147968 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:57:00.148077 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:57:00.148582 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 09:57:00.148598 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 09:57:00.151470 (+ 2872us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:34237: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":236,"threads_started":1}
W20260924 09:57:00.151858 9368 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 09:57:00.143818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:34445 (local address 127.0.97.62:34237)
0924 09:57:00.143974 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0924 09:57:00.143980 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 09:57:00.144356 (+ 376us) server_negotiation.cc:407] Connection header received
0924 09:57:00.144516 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:57:00.144520 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:57:00.144582 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:57:00.144655 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:57:00.145432 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:57:00.146131 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:57:00.147375 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:57:00.147832 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:57:00.147919 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:57:00.151656 (+ 3737us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:34445: BlockingRecv error: recv got EOF from 127.0.97.3:34445 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 24 09:57:00 dist-test-slave-w8d2 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 09:57:00 dist-test-slave-w8d2 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 09:57:00.310879 9673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:57:00.303394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34237 (local address 127.0.97.1:52435)
0924 09:57:00.303823 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0924 09:57:00.303838 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 09:57:00.303947 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:57:00.304211 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:57:00.304214 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:57:00.304547 (+ 333us) client_negotiation.cc:189] Negotiated authn=SASL
0924 09:57:00.304708 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:57:00.304716 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:57:00.305689 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:57:00.305693 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:57:00.306573 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:57:00.306583 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:57:00.307250 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:57:00.307253 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:57:00.307362 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:57:00.307904 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 09:57:00.307921 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 09:57:00.310638 (+ 2717us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:34237: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":241,"threads_started":1}
W20260924 09:57:00.310997 9368 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 09:57:00.303458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:52435 (local address 127.0.97.62:34237)
0924 09:57:00.303662 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0924 09:57:00.303667 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 09:57:00.303917 (+ 250us) server_negotiation.cc:407] Connection header received
0924 09:57:00.304034 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:57:00.304038 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:57:00.304099 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:57:00.304176 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:57:00.304811 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:57:00.305586 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:57:00.306722 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:57:00.307128 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:57:00.307217 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:57:00.310814 (+ 3597us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:52435: BlockingRecv error: recv got EOF from 127.0.97.1:52435 (error 108)
Metrics: {"server-negotiator.queue_time_us":108}
Sep 24 09:57:00 dist-test-slave-w8d2 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 09:57:00 dist-test-slave-w8d2 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 09:57:00.730377 9674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:57:00.721083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34237 (local address 127.0.97.2:51729)
0924 09:57:00.721538 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0924 09:57:00.721570 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0924 09:57:00.721701 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:57:00.722000 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:57:00.722005 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:57:00.722366 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0924 09:57:00.722529 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:57:00.722538 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:57:00.723606 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:57:00.723611 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:57:00.724779 (+ 1168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:57:00.724793 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:57:00.726729 (+ 1936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:57:00.726735 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:57:00.726892 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:57:00.727578 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 09:57:00.727609 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0924 09:57:00.730134 (+ 2525us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:34237: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":241,"threads_started":1}
W20260924 09:57:00.730472 9368 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 09:57:00.721250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:51729 (local address 127.0.97.62:34237)
0924 09:57:00.721459 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0924 09:57:00.721465 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 09:57:00.721648 (+ 183us) server_negotiation.cc:407] Connection header received
0924 09:57:00.721809 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:57:00.721814 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:57:00.721875 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:57:00.721955 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:57:00.722673 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:57:00.723462 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:57:00.726057 (+ 2595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:57:00.726591 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:57:00.726683 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:57:00.730305 (+ 3622us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:51729: BlockingRecv error: recv got EOF from 127.0.97.2:51729 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
Sep 24 09:57:01 dist-test-slave-w8d2 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 09:57:01 dist-test-slave-w8d2 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 09:57:01.160686 9675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:57:01.152667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34237 (local address 127.0.97.3:40833)
0924 09:57:01.153123 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0924 09:57:01.153156 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0924 09:57:01.153271 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:57:01.153618 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:57:01.153623 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:57:01.153997 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0924 09:57:01.154150 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:57:01.154159 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:57:01.155541 (+ 1382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:57:01.155546 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:57:01.156461 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:57:01.156471 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:57:01.157084 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:57:01.157089 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:57:01.157204 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:57:01.157738 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 09:57:01.157755 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 09:57:01.160436 (+ 2681us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:34237: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":260,"threads_started":1}
W20260924 09:57:01.160820 9368 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 09:57:01.152764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:40833 (local address 127.0.97.62:34237)
0924 09:57:01.152931 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0924 09:57:01.152938 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 09:57:01.153234 (+ 296us) server_negotiation.cc:407] Connection header received
0924 09:57:01.153371 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:57:01.153376 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:57:01.153477 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:57:01.153568 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:57:01.154285 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:57:01.155385 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:57:01.156577 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:57:01.156973 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:57:01.157066 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:57:01.160635 (+ 3569us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:40833: BlockingRecv error: recv got EOF from 127.0.97.3:40833 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Sep 24 09:57:01 dist-test-slave-w8d2 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 09:57:01 dist-test-slave-w8d2 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 09:57:01.319561 9676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:57:01.311954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34237 (local address 127.0.97.1:36343)
0924 09:57:01.312461 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0924 09:57:01.312477 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 09:57:01.312591 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:57:01.312842 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:57:01.312845 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:57:01.313217 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0924 09:57:01.313394 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:57:01.313404 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:57:01.314295 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:57:01.314298 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:57:01.315208 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:57:01.315219 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:57:01.316043 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:57:01.316048 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:57:01.316189 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:57:01.316755 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 09:57:01.316772 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 09:57:01.319316 (+ 2544us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:34237: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":293,"threads_started":1}
W20260924 09:57:01.319660 9368 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 09:57:01.312096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:36343 (local address 127.0.97.62:34237)
0924 09:57:01.312285 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0924 09:57:01.312289 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 09:57:01.312538 (+ 249us) server_negotiation.cc:407] Connection header received
0924 09:57:01.312677 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:57:01.312680 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:57:01.312736 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:57:01.312812 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:57:01.313513 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:57:01.314177 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:57:01.315369 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:57:01.315912 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:57:01.316026 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:57:01.319480 (+ 3454us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:36343: BlockingRecv error: recv got EOF from 127.0.97.1:36343 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 24 09:57:01 dist-test-slave-w8d2 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 09:57:01 dist-test-slave-w8d2 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 09:57:01.740326 9677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:57:01.731399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34237 (local address 127.0.97.2:43545)
0924 09:57:01.731925 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0924 09:57:01.731941 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 09:57:01.732072 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:57:01.732387 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:57:01.732390 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:57:01.732772 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0924 09:57:01.732986 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:57:01.732996 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:57:01.734422 (+ 1426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:57:01.734426 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:57:01.735584 (+ 1158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:57:01.735596 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:57:01.736522 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:57:01.736528 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:57:01.736687 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:57:01.737296 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 09:57:01.737313 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 09:57:01.740058 (+ 2745us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:34237: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":277,"threads_started":1}
W20260924 09:57:01.740736 9368 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 09:57:01.731486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:43545 (local address 127.0.97.62:34237)
0924 09:57:01.731684 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0924 09:57:01.731690 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 09:57:01.732014 (+ 324us) server_negotiation.cc:407] Connection header received
0924 09:57:01.732176 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:57:01.732181 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:57:01.732263 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:57:01.732375 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:57:01.733116 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:57:01.734283 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:57:01.735699 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:57:01.736392 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:57:01.736514 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:57:01.740296 (+ 3782us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:43545: BlockingRecv error: recv got EOF from 127.0.97.2:43545 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 09:57:02.005435 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu pid 9219
I20260924 09:57:02.016515 9679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:57:02.006392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55538 (local address 127.0.97.1:34019)
0924 09:57:02.006884 (+ 492us) server_negotiation.cc:206] Beginning negotiation
0924 09:57:02.006891 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 09:57:02.006916 (+ 25us) server_negotiation.cc:407] Connection header received
0924 09:57:02.006979 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:57:02.006985 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:57:02.007190 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:57:02.007394 (+ 204us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:57:02.008773 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:57:02.009742 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:57:02.010919 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:57:02.011543 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:57:02.011634 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:57:02.013799 (+ 2165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:57:02.013807 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:57:02.013822 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:57:02.013859 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:57:02.016074 (+ 2215us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.1@KRBTEST.COM not found in keytab (ticket kvno 2)
0924 09:57:02.016300 (+ 226us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:55538: Request ticket server kudu/127.0.97.1@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":343,"thread_start_us":213,"threads_started":1}
W20260924 09:57:02.016613 9679 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:55538: Request ticket server kudu/127.0.97.1@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260924 09:57:02.016711 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu pid 9219: Not authorized: Client connection negotiation failed: client connection to 127.0.97.1:34019: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.1@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260924 09:57:02.016901 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu with pid 9219
I20260924 09:57:02.030117 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu pid 9370
I20260924 09:57:02.040833 9680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:57:02.030990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38994 (local address 127.0.97.2:34931)
0924 09:57:02.031522 (+ 532us) server_negotiation.cc:206] Beginning negotiation
0924 09:57:02.031529 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 09:57:02.031558 (+ 29us) server_negotiation.cc:407] Connection header received
0924 09:57:02.031617 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:57:02.031624 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:57:02.031832 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:57:02.031972 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:57:02.033073 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:57:02.034082 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:57:02.035464 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:57:02.035999 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:57:02.036092 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:57:02.038198 (+ 2106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:57:02.038206 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:57:02.038219 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:57:02.038252 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:57:02.040376 (+ 2124us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.2@KRBTEST.COM not found in keytab (ticket kvno 2)
0924 09:57:02.040604 (+ 228us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:38994: Request ticket server kudu/127.0.97.2@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":387,"thread_start_us":256,"threads_started":1}
W20260924 09:57:02.040927 9680 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:38994: Request ticket server kudu/127.0.97.2@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260924 09:57:02.040951 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu pid 9370: Not authorized: Client connection negotiation failed: client connection to 127.0.97.2:34931: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.2@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260924 09:57:02.041127 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu with pid 9370
I20260924 09:57:02.054373 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu pid 9501
I20260924 09:57:02.065186 9681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:57:02.055216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41766 (local address 127.0.97.3:46587)
0924 09:57:02.055711 (+ 495us) server_negotiation.cc:206] Beginning negotiation
0924 09:57:02.055719 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 09:57:02.055745 (+ 26us) server_negotiation.cc:407] Connection header received
0924 09:57:02.055809 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:57:02.055816 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:57:02.056026 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:57:02.056176 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:57:02.057327 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:57:02.058345 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:57:02.059474 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:57:02.060030 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:57:02.060119 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:57:02.062322 (+ 2203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:57:02.062330 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:57:02.062343 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:57:02.062376 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:57:02.064707 (+ 2331us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.3@KRBTEST.COM not found in keytab (ticket kvno 2)
0924 09:57:02.064944 (+ 237us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:41766: Request ticket server kudu/127.0.97.3@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":360,"thread_start_us":239,"threads_started":1}
W20260924 09:57:02.065294 9681 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:41766: Request ticket server kudu/127.0.97.3@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260924 09:57:02.065337 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu pid 9501: Not authorized: Client connection negotiation failed: client connection to 127.0.97.3:46587: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.3@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260924 09:57:02.065485 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu with pid 9501
I20260924 09:57:02.077819 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu pid 9150
I20260924 09:57:02.078922 9212 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 09:57:02.188820 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu with pid 9150
2026-09-24T09:57:02Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (20540 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-24T09:57:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T09:57:02Z Disabled control of system clock
I20260924 09:57:02.252429 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
/tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790243735634835-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790243735634835-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790243735634835-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790243735634835-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44393
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:42679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:44393
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260924 09:57:02.554188 9686 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 09:57:02.554481 9686 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 09:57:02.554596 9686 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 09:57:02.561671 9686 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 09:57:02.561785 9686 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 09:57:02.561856 9686 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 09:57:02.561931 9686 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 09:57:02.561986 9686 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 09:57:02.570807 9686 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790243735634835-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790243735634835-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:44393
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44393
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790243735634835-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790243735634835-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 09:57:02.572064 9686 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 09:57:02.573704 9686 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 09:57:02.584426 9691 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:57:02.584421 9694 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:57:02.584455 9692 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 09:57:02.585351 9686 server_base.cc:1034] running on GCE node
I20260924 09:57:02.585988 9686 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 09:57:02.587204 9686 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 09:57:02.588476 9686 hybrid_clock.cc:648] HybridClock initialized: now 1790243822588426 us; error 45 us; skew 500 ppm
I20260924 09:57:02.588798 9686 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 09:57:02.590797 9686 webserver.cc:460] Webserver started at http://127.0.97.62:44139/ using document root <none> and password file <none>
I20260924 09:57:02.591451 9686 fs_manager.cc:359] Metadata directory not provided
I20260924 09:57:02.591542 9686 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 09:57:02.591835 9686 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 09:57:02.594507 9686 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790243735634835-388-0/minicluster-data/master-0/data/instance:
uuid: "c076ed8431e84d0aa4644049ce54d297"
format_stamp: "Formatted at 2026-09-24 09:57:02 on dist-test-slave-w8d2"
server_key: "b86b0d0ac131431a69f5e91a764962e8"
server_key_iv: "9a9665d9544ea3a3ba2ca09e6d26a7e4"
server_key_version: "encryptionkey@0"
I20260924 09:57:02.595278 9686 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790243735634835-388-0/minicluster-data/master-0/wal/instance:
uuid: "c076ed8431e84d0aa4644049ce54d297"
format_stamp: "Formatted at 2026-09-24 09:57:02 on dist-test-slave-w8d2"
server_key: "b86b0d0ac131431a69f5e91a764962e8"
server_key_iv: "9a9665d9544ea3a3ba2ca09e6d26a7e4"
server_key_version: "encryptionkey@0"
I20260924 09:57:02.599951 9686 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.002s
I20260924 09:57:02.603012 9700 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 09:57:02.604233 9686 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260924 09:57:02.604408 9686 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790243735634835-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790243735634835-388-0/minicluster-data/master-0/wal
uuid: "c076ed8431e84d0aa4644049ce54d297"
format_stamp: "Formatted at 2026-09-24 09:57:02 on dist-test-slave-w8d2"
server_key: "b86b0d0ac131431a69f5e91a764962e8"
server_key_iv: "9a9665d9544ea3a3ba2ca09e6d26a7e4"
server_key_version: "encryptionkey@0"
I20260924 09:57:02.604585 9686 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790243735634835-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790243735634835-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790243735634835-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 09:57:02.622186 9686 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 09:57:02.622929 9686 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 09:57:02.623169 9686 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 09:57:02.639334 9686 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:44393
I20260924 09:57:02.639397 9751 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:44393 every 8 connection(s)
I20260924 09:57:02.640558 9686 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790243735634835-388-0/minicluster-data/master-0/data/info.pb
I20260924 09:57:02.643977 9752 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 09:57:02.648487 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu as pid 9686
I20260924 09:57:02.648693 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790243735634835-388-0/minicluster-data/master-0/wal/instance
I20260924 09:57:02.649478 388 external_mini_cluster.cc:1428] Setting key 92412720eb1b693043dfc3305c6348c2
I20260924 09:57:02.655829 9752 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c076ed8431e84d0aa4644049ce54d297: Bootstrap starting.
I20260924 09:57:02.659372 9752 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c076ed8431e84d0aa4644049ce54d297: Neither blocks nor log segments found. Creating new log.
I20260924 09:57:02.660334 9752 log.cc:824] T 00000000000000000000000000000000 P c076ed8431e84d0aa4644049ce54d297: Log is configured to *not* fsync() on all Append() calls
I20260924 09:57:02.663316 9752 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c076ed8431e84d0aa4644049ce54d297: No bootstrap required, opened a new log
I20260924 09:57:02.664103 9755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:57:02.651459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56520 (local address 127.0.97.62:44393)
0924 09:57:02.652465 (+ 1006us) server_negotiation.cc:206] Beginning negotiation
0924 09:57:02.652479 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 09:57:02.652525 (+ 46us) server_negotiation.cc:407] Connection header received
0924 09:57:02.654784 (+ 2259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:57:02.654820 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:57:02.655385 (+ 565us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:57:02.655792 (+ 407us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:57:02.656312 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:57:02.658297 (+ 1985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:57:02.659607 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:57:02.660255 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:57:02.660413 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:57:02.661502 (+ 1089us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:57:02.661512 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:57:02.661529 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 09:57:02.662149 (+ 620us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:57:02.662240 (+ 91us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 09:57:02.662270 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:57:02.662382 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0924 09:57:02.662559 (+ 177us) server_negotiation.cc:299] Negotiation successful
0924 09:57:02.662860 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":520,"thread_start_us":310,"threads_started":1}
I20260924 09:57:02.670383 9752 raft_consensus.cc:348] T 00000000000000000000000000000000 P c076ed8431e84d0aa4644049ce54d297 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c076ed8431e84d0aa4644049ce54d297" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44393 } }
I20260924 09:57:02.670672 9752 raft_consensus.cc:374] T 00000000000000000000000000000000 P c076ed8431e84d0aa4644049ce54d297 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 09:57:02.670738 9752 raft_consensus.cc:729] T 00000000000000000000000000000000 P c076ed8431e84d0aa4644049ce54d297 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c076ed8431e84d0aa4644049ce54d297, State: Initialized, Role: FOLLOWER
I20260924 09:57:02.671202 9752 consensus_queue.cc:260] T 00000000000000000000000000000000 P c076ed8431e84d0aa4644049ce54d297 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c076ed8431e84d0aa4644049ce54d297" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44393 } }
I20260924 09:57:02.671420 9752 raft_consensus.cc:388] T 00000000000000000000000000000000 P c076ed8431e84d0aa4644049ce54d297 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 09:57:02.671511 9752 raft_consensus.cc:482] T 00000000000000000000000000000000 P c076ed8431e84d0aa4644049ce54d297 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 09:57:02.671628 9752 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c076ed8431e84d0aa4644049ce54d297 [term 0 FOLLOWER]: Advancing to term 1
I20260924 09:57:02.673532 9752 raft_consensus.cc:504] T 00000000000000000000000000000000 P c076ed8431e84d0aa4644049ce54d297 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c076ed8431e84d0aa4644049ce54d297" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44393 } }
I20260924 09:57:02.673921 9752 leader_election.cc:302] T 00000000000000000000000000000000 P c076ed8431e84d0aa4644049ce54d297 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c076ed8431e84d0aa4644049ce54d297; no voters:
I20260924 09:57:02.674376 9752 leader_election.cc:288] T 00000000000000000000000000000000 P c076ed8431e84d0aa4644049ce54d297 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 09:57:02.674515 9757 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c076ed8431e84d0aa4644049ce54d297 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 09:57:02.675148 9757 raft_consensus.cc:686] T 00000000000000000000000000000000 P c076ed8431e84d0aa4644049ce54d297 [term 1 LEADER]: Becoming Leader. State: Replica: c076ed8431e84d0aa4644049ce54d297, State: Running, Role: LEADER
I20260924 09:57:02.675638 9757 consensus_queue.cc:237] T 00000000000000000000000000000000 P c076ed8431e84d0aa4644049ce54d297 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c076ed8431e84d0aa4644049ce54d297" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44393 } }
I20260924 09:57:02.676331 9752 sys_catalog.cc:564] T 00000000000000000000000000000000 P c076ed8431e84d0aa4644049ce54d297 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 09:57:02.678063 9759 sys_catalog.cc:455] T 00000000000000000000000000000000 P c076ed8431e84d0aa4644049ce54d297 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c076ed8431e84d0aa4644049ce54d297. Latest consensus state: current_term: 1 leader_uuid: "c076ed8431e84d0aa4644049ce54d297" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c076ed8431e84d0aa4644049ce54d297" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44393 } } }
I20260924 09:57:02.678419 9759 sys_catalog.cc:458] T 00000000000000000000000000000000 P c076ed8431e84d0aa4644049ce54d297 [sys.catalog]: This master's current role is: LEADER
I20260924 09:57:02.678417 9758 sys_catalog.cc:455] T 00000000000000000000000000000000 P c076ed8431e84d0aa4644049ce54d297 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c076ed8431e84d0aa4644049ce54d297" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c076ed8431e84d0aa4644049ce54d297" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44393 } } }
I20260924 09:57:02.680048 9758 sys_catalog.cc:458] T 00000000000000000000000000000000 P c076ed8431e84d0aa4644049ce54d297 [sys.catalog]: This master's current role is: LEADER
I20260924 09:57:02.680409 9765 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 09:57:02.685783 9765 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 09:57:02.693807 9765 catalog_manager.cc:1347] Generated new cluster ID: 10036803bc9047b1a8ce768ee3edfa91
I20260924 09:57:02.693926 9765 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 09:57:02.726737 9765 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 09:57:02.728150 9765 catalog_manager.cc:1504] Loading token signing keys...
I20260924 09:57:02.735459 9765 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c076ed8431e84d0aa4644049ce54d297: Generated new TSK 0
I20260924 09:57:02.736351 9765 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 09:57:02.746403 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
/tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790243735634835-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790243735634835-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790243735634835-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790243735634835-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44393
--builtin_ntp_servers=127.0.97.20:42679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260924 09:57:03.053351 9776 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 09:57:03.053609 9776 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 09:57:03.053725 9776 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 09:57:03.060786 9776 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 09:57:03.060896 9776 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 09:57:03.061044 9776 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260924 09:57:03.069983 9776 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790243735634835-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790243735634835-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790243735634835-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44393
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790243735634835-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 09:57:03.071192 9776 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 09:57:03.072886 9776 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 09:57:03.084921 9784 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:57:03.084969 9782 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:57:03.085141 9781 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 09:57:03.085495 9776 server_base.cc:1034] running on GCE node
I20260924 09:57:03.086095 9776 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 09:57:03.086918 9776 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 09:57:03.088100 9776 hybrid_clock.cc:648] HybridClock initialized: now 1790243823088048 us; error 46 us; skew 500 ppm
I20260924 09:57:03.088416 9776 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 09:57:03.090452 9776 webserver.cc:460] Webserver started at http://127.0.97.1:45445/ using document root <none> and password file <none>
I20260924 09:57:03.091068 9776 fs_manager.cc:359] Metadata directory not provided
I20260924 09:57:03.091162 9776 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 09:57:03.091490 9776 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 09:57:03.094075 9776 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790243735634835-388-0/minicluster-data/ts-0/data/instance:
uuid: "969bf5bf37a54df586343b6805ee634a"
format_stamp: "Formatted at 2026-09-24 09:57:03 on dist-test-slave-w8d2"
server_key: "442f5a2e188c9e8d63aa19d9257a008a"
server_key_iv: "2dc293376cd16e1d14ea70e549dd5ea9"
server_key_version: "encryptionkey@0"
I20260924 09:57:03.094776 9776 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790243735634835-388-0/minicluster-data/ts-0/wal/instance:
uuid: "969bf5bf37a54df586343b6805ee634a"
format_stamp: "Formatted at 2026-09-24 09:57:03 on dist-test-slave-w8d2"
server_key: "442f5a2e188c9e8d63aa19d9257a008a"
server_key_iv: "2dc293376cd16e1d14ea70e549dd5ea9"
server_key_version: "encryptionkey@0"
I20260924 09:57:03.099511 9776 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260924 09:57:03.102471 9790 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 09:57:03.103708 9776 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 09:57:03.103843 9776 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790243735634835-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790243735634835-388-0/minicluster-data/ts-0/wal
uuid: "969bf5bf37a54df586343b6805ee634a"
format_stamp: "Formatted at 2026-09-24 09:57:03 on dist-test-slave-w8d2"
server_key: "442f5a2e188c9e8d63aa19d9257a008a"
server_key_iv: "2dc293376cd16e1d14ea70e549dd5ea9"
server_key_version: "encryptionkey@0"
I20260924 09:57:03.104017 9776 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790243735634835-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790243735634835-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790243735634835-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 09:57:03.117144 9776 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 09:57:03.117875 9776 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 09:57:03.118109 9776 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 09:57:03.118921 9776 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 09:57:03.120292 9776 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 09:57:03.120391 9776 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 09:57:03.120492 9776 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 09:57:03.120558 9776 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 09:57:03.148108 9776 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:45991
I20260924 09:57:03.148147 9902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:45991 every 8 connection(s)
I20260924 09:57:03.149300 9776 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790243735634835-388-0/minicluster-data/ts-0/data/info.pb
I20260924 09:57:03.157361 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu as pid 9776
I20260924 09:57:03.157562 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790243735634835-388-0/minicluster-data/ts-0/wal/instance
I20260924 09:57:03.158252 388 external_mini_cluster.cc:1428] Setting key 6e05700432a6b4a7498033f30f502aa0
I20260924 09:57:03.160627 9755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:57:03.151615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:35499 (local address 127.0.97.62:44393)
0924 09:57:03.151799 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0924 09:57:03.151805 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 09:57:03.153078 (+ 1273us) server_negotiation.cc:407] Connection header received
0924 09:57:03.154634 (+ 1556us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:57:03.154638 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:57:03.154700 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:57:03.154782 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:57:03.155736 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:57:03.156417 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:57:03.157918 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:57:03.158510 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:57:03.158633 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:57:03.159890 (+ 1257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:57:03.159895 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:57:03.159898 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 09:57:03.159915 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:57:03.159943 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 09:57:03.159955 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:57:03.160027 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 09:57:03.160260 (+ 233us) server_negotiation.cc:299] Negotiation successful
0924 09:57:03.160343 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 09:57:03.160964 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
/tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790243735634835-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790243735634835-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790243735634835-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790243735634835-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44393
--builtin_ntp_servers=127.0.97.20:42679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260924 09:57:03.161576 9905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:57:03.152113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44393 (local address 127.0.97.1:35499)
0924 09:57:03.152938 (+ 825us) negotiation.cc:107] Waiting for socket to connect
0924 09:57:03.152976 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 09:57:03.154410 (+ 1434us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:57:03.154955 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:57:03.154962 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:57:03.155039 (+ 77us) client_negotiation.cc:189] Negotiated authn=SASL
0924 09:57:03.155610 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:57:03.155626 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:57:03.156535 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:57:03.156539 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:57:03.157781 (+ 1242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:57:03.157792 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:57:03.158689 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:57:03.158694 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:57:03.158900 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:57:03.159599 (+ 699us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 09:57:03.159601 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 09:57:03.159638 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 09:57:03.159706 (+ 68us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 09:57:03.160045 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 09:57:03.160056 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 09:57:03.160064 (+ 8us) client_negotiation.cc:769] Sending connection context
0924 09:57:03.160222 (+ 158us) client_negotiation.cc:240] Negotiation successful
0924 09:57:03.160481 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":253,"threads_started":1}
I20260924 09:57:03.163010 9903 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44393
I20260924 09:57:03.163283 9903 heartbeater.cc:464] Registering TS with master...
I20260924 09:57:03.163962 9903 heartbeater.cc:511] Master 127.0.97.62:44393 requested a full tablet report, sending...
I20260924 09:57:03.165738 9717 ts_manager.cc:194] Registered new tserver with Master: 969bf5bf37a54df586343b6805ee634a (127.0.97.1:45991)
I20260924 09:57:03.167114 9717 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.1:35499
W20260924 09:57:03.468649 9906 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 09:57:03.468920 9906 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 09:57:03.469020 9906 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 09:57:03.476109 9906 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 09:57:03.476209 9906 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 09:57:03.476346 9906 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260924 09:57:03.485946 9906 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790243735634835-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790243735634835-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790243735634835-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44393
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790243735634835-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 09:57:03.487214 9906 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 09:57:03.488977 9906 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 09:57:03.501387 9912 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:57:03.501360 9911 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:57:03.501431 9914 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 09:57:03.502008 9906 server_base.cc:1034] running on GCE node
I20260924 09:57:03.502600 9906 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 09:57:03.503535 9906 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 09:57:03.504729 9906 hybrid_clock.cc:648] HybridClock initialized: now 1790243823504676 us; error 46 us; skew 500 ppm
I20260924 09:57:03.505020 9906 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 09:57:03.507073 9906 webserver.cc:460] Webserver started at http://127.0.97.2:35285/ using document root <none> and password file <none>
I20260924 09:57:03.507893 9906 fs_manager.cc:359] Metadata directory not provided
I20260924 09:57:03.507993 9906 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 09:57:03.508306 9906 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 09:57:03.511106 9906 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790243735634835-388-0/minicluster-data/ts-1/data/instance:
uuid: "ab51b16d11d047e591f67963ae8fff4d"
format_stamp: "Formatted at 2026-09-24 09:57:03 on dist-test-slave-w8d2"
server_key: "7950397abd9aaefbc2095add5441f940"
server_key_iv: "0c0e77877b5c530199821f209d2d9aed"
server_key_version: "encryptionkey@0"
I20260924 09:57:03.511924 9906 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790243735634835-388-0/minicluster-data/ts-1/wal/instance:
uuid: "ab51b16d11d047e591f67963ae8fff4d"
format_stamp: "Formatted at 2026-09-24 09:57:03 on dist-test-slave-w8d2"
server_key: "7950397abd9aaefbc2095add5441f940"
server_key_iv: "0c0e77877b5c530199821f209d2d9aed"
server_key_version: "encryptionkey@0"
I20260924 09:57:03.516830 9906 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260924 09:57:03.520077 9920 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 09:57:03.521314 9906 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260924 09:57:03.521468 9906 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790243735634835-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790243735634835-388-0/minicluster-data/ts-1/wal
uuid: "ab51b16d11d047e591f67963ae8fff4d"
format_stamp: "Formatted at 2026-09-24 09:57:03 on dist-test-slave-w8d2"
server_key: "7950397abd9aaefbc2095add5441f940"
server_key_iv: "0c0e77877b5c530199821f209d2d9aed"
server_key_version: "encryptionkey@0"
I20260924 09:57:03.521675 9906 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790243735634835-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790243735634835-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790243735634835-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 09:57:03.536691 9906 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 09:57:03.537501 9906 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 09:57:03.537746 9906 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 09:57:03.538619 9906 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 09:57:03.540062 9906 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 09:57:03.540155 9906 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 09:57:03.540261 9906 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 09:57:03.540328 9906 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 09:57:03.575189 9906 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:37557
I20260924 09:57:03.575255 10032 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:37557 every 8 connection(s)
I20260924 09:57:03.576807 9906 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790243735634835-388-0/minicluster-data/ts-1/data/info.pb
I20260924 09:57:03.585989 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu as pid 9906
I20260924 09:57:03.586149 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790243735634835-388-0/minicluster-data/ts-1/wal/instance
I20260924 09:57:03.586570 388 external_mini_cluster.cc:1428] Setting key 537a135097b084d1e82370f77e6bd36a
I20260924 09:57:03.589143 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
/tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790243735634835-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790243735634835-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790243735634835-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790243735634835-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44393
--builtin_ntp_servers=127.0.97.20:42679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260924 09:57:03.591325 9755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:57:03.579653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:50477 (local address 127.0.97.62:44393)
0924 09:57:03.579861 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0924 09:57:03.579868 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 09:57:03.581490 (+ 1622us) server_negotiation.cc:407] Connection header received
0924 09:57:03.583913 (+ 2423us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:57:03.583919 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:57:03.584015 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:57:03.584137 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:57:03.585499 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:57:03.586598 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:57:03.588765 (+ 2167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:57:03.589344 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:57:03.589465 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:57:03.590626 (+ 1161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:57:03.590631 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:57:03.590636 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 09:57:03.590657 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:57:03.590691 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 09:57:03.590704 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:57:03.590780 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 09:57:03.590949 (+ 169us) server_negotiation.cc:299] Negotiation successful
0924 09:57:03.591030 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260924 09:57:03.592280 10035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:57:03.580185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44393 (local address 127.0.97.2:50477)
0924 09:57:03.581318 (+ 1133us) negotiation.cc:107] Waiting for socket to connect
0924 09:57:03.581368 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0924 09:57:03.583586 (+ 2218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:57:03.584406 (+ 820us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:57:03.584419 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:57:03.584541 (+ 122us) client_negotiation.cc:189] Negotiated authn=SASL
0924 09:57:03.585316 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:57:03.585341 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:57:03.586833 (+ 1492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:57:03.586838 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:57:03.588631 (+ 1793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:57:03.588643 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:57:03.589521 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:57:03.589526 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:57:03.589716 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:57:03.590345 (+ 629us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 09:57:03.590347 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 09:57:03.590388 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 09:57:03.590465 (+ 77us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 09:57:03.590790 (+ 325us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 09:57:03.590795 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 09:57:03.590801 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 09:57:03.590926 (+ 125us) client_negotiation.cc:240] Negotiation successful
0924 09:57:03.591224 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":487,"thread_start_us":288,"threads_started":1}
I20260924 09:57:03.593523 10033 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44393
I20260924 09:57:03.593782 10033 heartbeater.cc:464] Registering TS with master...
I20260924 09:57:03.594467 10033 heartbeater.cc:511] Master 127.0.97.62:44393 requested a full tablet report, sending...
I20260924 09:57:03.595896 9717 ts_manager.cc:194] Registered new tserver with Master: ab51b16d11d047e591f67963ae8fff4d (127.0.97.2:37557)
I20260924 09:57:03.596943 9717 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.2:50477
W20260924 09:57:03.896324 10036 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 09:57:03.896677 10036 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 09:57:03.896807 10036 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 09:57:03.903888 10036 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 09:57:03.903985 10036 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 09:57:03.904122 10036 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260924 09:57:03.913035 10036 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790243735634835-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790243735634835-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790243735634835-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44393
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790243735634835-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 09:57:03.914229 10036 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 09:57:03.915927 10036 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 09:57:03.927901 10044 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:57:03.927942 10041 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:57:03.927970 10042 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 09:57:03.928750 10036 server_base.cc:1034] running on GCE node
I20260924 09:57:03.929322 10036 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 09:57:03.930207 10036 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 09:57:03.931381 10036 hybrid_clock.cc:648] HybridClock initialized: now 1790243823931332 us; error 43 us; skew 500 ppm
I20260924 09:57:03.931666 10036 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 09:57:03.933642 10036 webserver.cc:460] Webserver started at http://127.0.97.3:40023/ using document root <none> and password file <none>
I20260924 09:57:03.934278 10036 fs_manager.cc:359] Metadata directory not provided
I20260924 09:57:03.934374 10036 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 09:57:03.934679 10036 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 09:57:03.937388 10036 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790243735634835-388-0/minicluster-data/ts-2/data/instance:
uuid: "8ef67e2f438b4fcc9bd44d48a567c9d2"
format_stamp: "Formatted at 2026-09-24 09:57:03 on dist-test-slave-w8d2"
server_key: "e264b243361c840a8242dd0c8c91b5ce"
server_key_iv: "963a465cb856945218d4dc36c5447373"
server_key_version: "encryptionkey@0"
I20260924 09:57:03.938107 10036 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790243735634835-388-0/minicluster-data/ts-2/wal/instance:
uuid: "8ef67e2f438b4fcc9bd44d48a567c9d2"
format_stamp: "Formatted at 2026-09-24 09:57:03 on dist-test-slave-w8d2"
server_key: "e264b243361c840a8242dd0c8c91b5ce"
server_key_iv: "963a465cb856945218d4dc36c5447373"
server_key_version: "encryptionkey@0"
I20260924 09:57:03.942830 10036 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.002s
I20260924 09:57:03.946337 10050 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 09:57:03.947659 10036 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 09:57:03.947808 10036 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790243735634835-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790243735634835-388-0/minicluster-data/ts-2/wal
uuid: "8ef67e2f438b4fcc9bd44d48a567c9d2"
format_stamp: "Formatted at 2026-09-24 09:57:03 on dist-test-slave-w8d2"
server_key: "e264b243361c840a8242dd0c8c91b5ce"
server_key_iv: "963a465cb856945218d4dc36c5447373"
server_key_version: "encryptionkey@0"
I20260924 09:57:03.947995 10036 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790243735634835-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790243735634835-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790243735634835-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 09:57:03.973599 10036 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 09:57:03.974444 10036 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 09:57:03.974710 10036 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 09:57:03.975636 10036 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 09:57:03.977032 10036 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 09:57:03.977124 10036 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 09:57:03.977223 10036 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 09:57:03.977289 10036 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 09:57:04.005251 10036 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:35091
I20260924 09:57:04.005306 10162 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:35091 every 8 connection(s)
I20260924 09:57:04.006513 10036 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790243735634835-388-0/minicluster-data/ts-2/data/info.pb
I20260924 09:57:04.010183 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu as pid 10036
I20260924 09:57:04.010363 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790243735634835-388-0/minicluster-data/ts-2/wal/instance
I20260924 09:57:04.010897 388 external_mini_cluster.cc:1428] Setting key c84e98691c36ae20a868f726a6bb9fe4
I20260924 09:57:04.018383 9755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:57:04.008951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:33415 (local address 127.0.97.62:44393)
0924 09:57:04.009116 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0924 09:57:04.009120 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 09:57:04.010670 (+ 1550us) server_negotiation.cc:407] Connection header received
0924 09:57:04.012743 (+ 2073us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:57:04.012748 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:57:04.012825 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:57:04.012907 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:57:04.014134 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:57:04.014817 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:57:04.016267 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:57:04.016646 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:57:04.016730 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:57:04.017825 (+ 1095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:57:04.017830 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:57:04.017833 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 09:57:04.017848 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:57:04.017882 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 09:57:04.017893 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:57:04.017958 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0924 09:57:04.018104 (+ 146us) server_negotiation.cc:299] Negotiation successful
0924 09:57:04.018172 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 09:57:04.019332 10165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:57:04.009459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44393 (local address 127.0.97.3:33415)
0924 09:57:04.010502 (+ 1043us) negotiation.cc:107] Waiting for socket to connect
0924 09:57:04.010549 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0924 09:57:04.012479 (+ 1930us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:57:04.013145 (+ 666us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:57:04.013157 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:57:04.013278 (+ 121us) client_negotiation.cc:189] Negotiated authn=SASL
0924 09:57:04.013994 (+ 716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:57:04.014012 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:57:04.014932 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:57:04.014936 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:57:04.016168 (+ 1232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:57:04.016179 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:57:04.016799 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:57:04.016803 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:57:04.016983 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:57:04.017561 (+ 578us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 09:57:04.017563 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 09:57:04.017610 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 09:57:04.017685 (+ 75us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 09:57:04.017959 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 09:57:04.017964 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 09:57:04.017969 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 09:57:04.018089 (+ 120us) client_negotiation.cc:240] Negotiation successful
0924 09:57:04.018351 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":269,"threads_started":1}
I20260924 09:57:04.020452 10163 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44393
I20260924 09:57:04.020714 10163 heartbeater.cc:464] Registering TS with master...
I20260924 09:57:04.021375 10163 heartbeater.cc:511] Master 127.0.97.62:44393 requested a full tablet report, sending...
I20260924 09:57:04.022557 9717 ts_manager.cc:194] Registered new tserver with Master: 8ef67e2f438b4fcc9bd44d48a567c9d2 (127.0.97.3:35091)
I20260924 09:57:04.023305 9717 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.3:33415
I20260924 09:57:04.025822 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260924 09:57:04.031279 9755 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 09:57:04.029810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36656 (local address 127.0.97.62:44393)
0924 09:57:04.029959 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0924 09:57:04.029963 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 09:57:04.030434 (+ 471us) server_negotiation.cc:407] Connection header received
0924 09:57:04.030562 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:57:04.030566 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:57:04.030662 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:57:04.030738 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:57:04.031011 (+ 273us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36656: BlockingRecv error: recv got EOF from 127.0.0.1:36656 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 09:57:04.031949 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu pid 9776
I20260924 09:57:04.038712 10172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:57:04.032767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48322 (local address 127.0.97.1:45991)
0924 09:57:04.033242 (+ 475us) server_negotiation.cc:206] Beginning negotiation
0924 09:57:04.033248 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 09:57:04.033272 (+ 24us) server_negotiation.cc:407] Connection header received
0924 09:57:04.033327 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:57:04.033334 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:57:04.033533 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:57:04.033653 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:57:04.033894 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:57:04.034829 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:57:04.036008 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:57:04.036543 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:57:04.036634 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:57:04.037450 (+ 816us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:57:04.037458 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:57:04.037471 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 09:57:04.038022 (+ 551us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:57:04.038067 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 09:57:04.038090 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:57:04.038172 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0924 09:57:04.038264 (+ 92us) server_negotiation.cc:299] Negotiation successful
0924 09:57:04.038386 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":242,"threads_started":1}
I20260924 09:57:04.039881 9898 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 09:57:04.118157 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu with pid 9776
I20260924 09:57:04.131300 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu pid 9906
I20260924 09:57:04.138571 10174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:57:04.132115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43570 (local address 127.0.97.2:37557)
0924 09:57:04.132599 (+ 484us) server_negotiation.cc:206] Beginning negotiation
0924 09:57:04.132606 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 09:57:04.132633 (+ 27us) server_negotiation.cc:407] Connection header received
0924 09:57:04.132695 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:57:04.132701 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:57:04.132905 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:57:04.133048 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:57:04.133417 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:57:04.134397 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:57:04.135688 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:57:04.136219 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:57:04.136329 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:57:04.137242 (+ 913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:57:04.137250 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:57:04.137263 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 09:57:04.137829 (+ 566us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:57:04.137877 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 09:57:04.137893 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:57:04.137984 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0924 09:57:04.138131 (+ 147us) server_negotiation.cc:299] Negotiation successful
0924 09:57:04.138271 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":239,"threads_started":1}
I20260924 09:57:04.139856 10028 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 09:57:04.219377 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu with pid 9906
I20260924 09:57:04.231863 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu pid 10036
I20260924 09:57:04.242750 10176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:57:04.232768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52334 (local address 127.0.97.3:35091)
0924 09:57:04.233363 (+ 595us) server_negotiation.cc:206] Beginning negotiation
0924 09:57:04.233369 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 09:57:04.233398 (+ 29us) server_negotiation.cc:407] Connection header received
0924 09:57:04.233452 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:57:04.233458 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:57:04.233677 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:57:04.233795 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:57:04.234183 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:57:04.235190 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:57:04.236457 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:57:04.236995 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:57:04.237090 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:57:04.241421 (+ 4331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:57:04.241429 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:57:04.241444 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 09:57:04.242009 (+ 565us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:57:04.242056 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 09:57:04.242074 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:57:04.242161 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0924 09:57:04.242312 (+ 151us) server_negotiation.cc:299] Negotiation successful
0924 09:57:04.242419 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":462,"thread_start_us":320,"threads_started":1}
I20260924 09:57:04.244009 10158 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 09:57:04.324018 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu with pid 10036
I20260924 09:57:04.336479 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu pid 9686
I20260924 09:57:04.337594 9747 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 09:57:04.436443 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu with pid 9686
2026-09-24T09:57:04Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2244 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-24T09:57:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T09:57:04Z Disabled control of system clock
I20260924 09:57:04.499378 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
/tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790243735634835-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790243735634835-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790243735634835-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790243735634835-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40515
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:39687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:40515
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260924 09:57:04.811369 10182 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 09:57:04.811669 10182 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 09:57:04.811790 10182 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 09:57:04.818923 10182 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 09:57:04.819029 10182 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 09:57:04.819078 10182 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 09:57:04.819156 10182 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 09:57:04.819217 10182 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 09:57:04.828401 10182 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790243735634835-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790243735634835-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:40515
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40515
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790243735634835-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790243735634835-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 09:57:04.829676 10182 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 09:57:04.831481 10182 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 09:57:04.842377 10187 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:57:04.842414 10188 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:57:04.842347 10190 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 09:57:04.843132 10182 server_base.cc:1034] running on GCE node
I20260924 09:57:04.843920 10182 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 09:57:04.845270 10182 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 09:57:04.846467 10182 hybrid_clock.cc:648] HybridClock initialized: now 1790243824846424 us; error 33 us; skew 500 ppm
I20260924 09:57:04.846801 10182 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 09:57:04.848925 10182 webserver.cc:460] Webserver started at http://127.0.97.62:44985/ using document root <none> and password file <none>
I20260924 09:57:04.849560 10182 fs_manager.cc:359] Metadata directory not provided
I20260924 09:57:04.849645 10182 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 09:57:04.849890 10182 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 09:57:04.852849 10182 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790243735634835-388-0/minicluster-data/master-0/data/instance:
uuid: "381ddb19387d4a1e8b320ba37fca8430"
format_stamp: "Formatted at 2026-09-24 09:57:04 on dist-test-slave-w8d2"
server_key: "4fa2024fee86ba49cd5462fb115e0c6d"
server_key_iv: "9703a7320cbc47b72570ba6184bd0132"
server_key_version: "encryptionkey@0"
I20260924 09:57:04.853578 10182 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790243735634835-388-0/minicluster-data/master-0/wal/instance:
uuid: "381ddb19387d4a1e8b320ba37fca8430"
format_stamp: "Formatted at 2026-09-24 09:57:04 on dist-test-slave-w8d2"
server_key: "4fa2024fee86ba49cd5462fb115e0c6d"
server_key_iv: "9703a7320cbc47b72570ba6184bd0132"
server_key_version: "encryptionkey@0"
I20260924 09:57:04.858878 10182 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260924 09:57:04.862188 10196 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 09:57:04.863544 10182 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260924 09:57:04.863773 10182 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790243735634835-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790243735634835-388-0/minicluster-data/master-0/wal
uuid: "381ddb19387d4a1e8b320ba37fca8430"
format_stamp: "Formatted at 2026-09-24 09:57:04 on dist-test-slave-w8d2"
server_key: "4fa2024fee86ba49cd5462fb115e0c6d"
server_key_iv: "9703a7320cbc47b72570ba6184bd0132"
server_key_version: "encryptionkey@0"
I20260924 09:57:04.864010 10182 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790243735634835-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790243735634835-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790243735634835-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 09:57:04.869467 10182 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 09:57:04.870193 10182 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 09:57:04.870465 10182 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 09:57:04.886590 10182 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:40515
I20260924 09:57:04.886607 10247 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:40515 every 8 connection(s)
I20260924 09:57:04.887912 10182 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790243735634835-388-0/minicluster-data/master-0/data/info.pb
I20260924 09:57:04.891563 10248 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 09:57:04.898118 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu as pid 10182
I20260924 09:57:04.898298 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790243735634835-388-0/minicluster-data/master-0/wal/instance
I20260924 09:57:04.898916 388 external_mini_cluster.cc:1428] Setting key 65882865c4ac9063e77e48d13b742647
I20260924 09:57:04.901654 10248 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 381ddb19387d4a1e8b320ba37fca8430: Bootstrap starting.
I20260924 09:57:04.905622 10248 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 381ddb19387d4a1e8b320ba37fca8430: Neither blocks nor log segments found. Creating new log.
I20260924 09:57:04.906930 10248 log.cc:824] T 00000000000000000000000000000000 P 381ddb19387d4a1e8b320ba37fca8430: Log is configured to *not* fsync() on all Append() calls
I20260924 09:57:04.908123 10251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:57:04.900981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42968 (local address 127.0.97.62:40515)
0924 09:57:04.902024 (+ 1043us) server_negotiation.cc:206] Beginning negotiation
0924 09:57:04.902039 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 09:57:04.902085 (+ 46us) server_negotiation.cc:407] Connection header received
0924 09:57:04.904320 (+ 2235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:57:04.904346 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:57:04.904867 (+ 521us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:57:04.905279 (+ 412us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:57:04.906293 (+ 1014us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:57:04.906301 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:57:04.906317 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 09:57:04.906334 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:57:04.906399 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 09:57:04.906423 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:57:04.906618 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0924 09:57:04.906747 (+ 129us) server_negotiation.cc:299] Negotiation successful
0924 09:57:04.907017 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":502,"thread_start_us":288,"threads_started":1}
I20260924 09:57:04.909709 10248 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 381ddb19387d4a1e8b320ba37fca8430: No bootstrap required, opened a new log
I20260924 09:57:04.917043 10248 raft_consensus.cc:348] T 00000000000000000000000000000000 P 381ddb19387d4a1e8b320ba37fca8430 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "381ddb19387d4a1e8b320ba37fca8430" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40515 } }
I20260924 09:57:04.917373 10248 raft_consensus.cc:374] T 00000000000000000000000000000000 P 381ddb19387d4a1e8b320ba37fca8430 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 09:57:04.917464 10248 raft_consensus.cc:729] T 00000000000000000000000000000000 P 381ddb19387d4a1e8b320ba37fca8430 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 381ddb19387d4a1e8b320ba37fca8430, State: Initialized, Role: FOLLOWER
I20260924 09:57:04.918025 10248 consensus_queue.cc:260] T 00000000000000000000000000000000 P 381ddb19387d4a1e8b320ba37fca8430 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "381ddb19387d4a1e8b320ba37fca8430" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40515 } }
I20260924 09:57:04.918232 10248 raft_consensus.cc:388] T 00000000000000000000000000000000 P 381ddb19387d4a1e8b320ba37fca8430 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 09:57:04.918349 10248 raft_consensus.cc:482] T 00000000000000000000000000000000 P 381ddb19387d4a1e8b320ba37fca8430 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 09:57:04.918494 10248 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 381ddb19387d4a1e8b320ba37fca8430 [term 0 FOLLOWER]: Advancing to term 1
I20260924 09:57:04.920679 10248 raft_consensus.cc:504] T 00000000000000000000000000000000 P 381ddb19387d4a1e8b320ba37fca8430 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "381ddb19387d4a1e8b320ba37fca8430" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40515 } }
I20260924 09:57:04.921139 10248 leader_election.cc:302] T 00000000000000000000000000000000 P 381ddb19387d4a1e8b320ba37fca8430 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 381ddb19387d4a1e8b320ba37fca8430; no voters:
I20260924 09:57:04.921633 10248 leader_election.cc:288] T 00000000000000000000000000000000 P 381ddb19387d4a1e8b320ba37fca8430 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 09:57:04.921773 10253 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 381ddb19387d4a1e8b320ba37fca8430 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 09:57:04.922381 10253 raft_consensus.cc:686] T 00000000000000000000000000000000 P 381ddb19387d4a1e8b320ba37fca8430 [term 1 LEADER]: Becoming Leader. State: Replica: 381ddb19387d4a1e8b320ba37fca8430, State: Running, Role: LEADER
I20260924 09:57:04.922928 10253 consensus_queue.cc:237] T 00000000000000000000000000000000 P 381ddb19387d4a1e8b320ba37fca8430 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "381ddb19387d4a1e8b320ba37fca8430" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40515 } }
I20260924 09:57:04.923590 10248 sys_catalog.cc:564] T 00000000000000000000000000000000 P 381ddb19387d4a1e8b320ba37fca8430 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 09:57:04.925669 10254 sys_catalog.cc:455] T 00000000000000000000000000000000 P 381ddb19387d4a1e8b320ba37fca8430 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "381ddb19387d4a1e8b320ba37fca8430" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "381ddb19387d4a1e8b320ba37fca8430" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40515 } } }
I20260924 09:57:04.926002 10254 sys_catalog.cc:458] T 00000000000000000000000000000000 P 381ddb19387d4a1e8b320ba37fca8430 [sys.catalog]: This master's current role is: LEADER
I20260924 09:57:04.926612 10255 sys_catalog.cc:455] T 00000000000000000000000000000000 P 381ddb19387d4a1e8b320ba37fca8430 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 381ddb19387d4a1e8b320ba37fca8430. Latest consensus state: current_term: 1 leader_uuid: "381ddb19387d4a1e8b320ba37fca8430" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "381ddb19387d4a1e8b320ba37fca8430" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40515 } } }
I20260924 09:57:04.927008 10255 sys_catalog.cc:458] T 00000000000000000000000000000000 P 381ddb19387d4a1e8b320ba37fca8430 [sys.catalog]: This master's current role is: LEADER
I20260924 09:57:04.927426 10260 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 09:57:04.932667 10260 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 09:57:04.941339 10260 catalog_manager.cc:1347] Generated new cluster ID: 5d9e14c5dec74d0d946db2b98d340bbc
I20260924 09:57:04.941437 10260 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 09:57:04.989696 10260 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 09:57:04.990127 10260 catalog_manager.cc:1504] Loading token signing keys...
I20260924 09:57:04.999039 10260 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 381ddb19387d4a1e8b320ba37fca8430: Generated new TSK 0
I20260924 09:57:04.999902 10260 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 09:57:05.007445 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
/tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790243735634835-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790243735634835-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790243735634835-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790243735634835-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40515
--builtin_ntp_servers=127.0.97.20:39687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260924 09:57:05.345391 10272 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 09:57:05.345678 10272 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 09:57:05.345808 10272 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 09:57:05.352978 10272 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 09:57:05.353087 10272 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 09:57:05.353224 10272 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260924 09:57:05.362301 10272 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790243735634835-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790243735634835-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790243735634835-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40515
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790243735634835-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 09:57:05.363597 10272 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 09:57:05.365293 10272 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 09:57:05.377538 10277 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:57:05.377573 10278 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:57:05.377609 10280 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 09:57:05.378396 10272 server_base.cc:1034] running on GCE node
I20260924 09:57:05.378986 10272 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 09:57:05.379945 10272 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 09:57:05.381143 10272 hybrid_clock.cc:648] HybridClock initialized: now 1790243825381094 us; error 42 us; skew 500 ppm
I20260924 09:57:05.381453 10272 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 09:57:05.383628 10272 webserver.cc:460] Webserver started at http://127.0.97.1:44717/ using document root <none> and password file <none>
I20260924 09:57:05.384289 10272 fs_manager.cc:359] Metadata directory not provided
I20260924 09:57:05.384382 10272 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 09:57:05.384667 10272 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 09:57:05.387428 10272 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790243735634835-388-0/minicluster-data/ts-0/data/instance:
uuid: "e270115e259c4a93a12a0ae52520026e"
format_stamp: "Formatted at 2026-09-24 09:57:05 on dist-test-slave-w8d2"
server_key: "76ee6c8170da6bfdc3008727fef4a39f"
server_key_iv: "c218105eb57c604b46befd7b9c021040"
server_key_version: "encryptionkey@0"
I20260924 09:57:05.388162 10272 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790243735634835-388-0/minicluster-data/ts-0/wal/instance:
uuid: "e270115e259c4a93a12a0ae52520026e"
format_stamp: "Formatted at 2026-09-24 09:57:05 on dist-test-slave-w8d2"
server_key: "76ee6c8170da6bfdc3008727fef4a39f"
server_key_iv: "c218105eb57c604b46befd7b9c021040"
server_key_version: "encryptionkey@0"
I20260924 09:57:05.393005 10272 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 09:57:05.396211 10286 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 09:57:05.397428 10272 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 09:57:05.397569 10272 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790243735634835-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790243735634835-388-0/minicluster-data/ts-0/wal
uuid: "e270115e259c4a93a12a0ae52520026e"
format_stamp: "Formatted at 2026-09-24 09:57:05 on dist-test-slave-w8d2"
server_key: "76ee6c8170da6bfdc3008727fef4a39f"
server_key_iv: "c218105eb57c604b46befd7b9c021040"
server_key_version: "encryptionkey@0"
I20260924 09:57:05.397765 10272 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790243735634835-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790243735634835-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790243735634835-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 09:57:05.402856 10272 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 09:57:05.403654 10272 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 09:57:05.403896 10272 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 09:57:05.404837 10272 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 09:57:05.406172 10272 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 09:57:05.406270 10272 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 09:57:05.406376 10272 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 09:57:05.406450 10272 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 09:57:05.434584 10272 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:40071
I20260924 09:57:05.434624 10398 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:40071 every 8 connection(s)
I20260924 09:57:05.435889 10272 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790243735634835-388-0/minicluster-data/ts-0/data/info.pb
I20260924 09:57:05.440877 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu as pid 10272
I20260924 09:57:05.441035 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790243735634835-388-0/minicluster-data/ts-0/wal/instance
I20260924 09:57:05.441607 388 external_mini_cluster.cc:1428] Setting key 5cc446ab5af041d7e92aad0dd4de89b5
I20260924 09:57:05.443652 10401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:57:05.438265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:57037 (local address 127.0.97.62:40515)
0924 09:57:05.438633 (+ 368us) server_negotiation.cc:206] Beginning negotiation
0924 09:57:05.438637 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 09:57:05.439852 (+ 1215us) server_negotiation.cc:407] Connection header received
0924 09:57:05.441473 (+ 1621us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:57:05.441478 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:57:05.441540 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:57:05.441631 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:57:05.442823 (+ 1192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:57:05.442828 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:57:05.442833 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 09:57:05.442851 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:57:05.442887 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 09:57:05.442899 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:57:05.442997 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0924 09:57:05.443308 (+ 311us) server_negotiation.cc:299] Negotiation successful
0924 09:57:05.443365 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":189,"threads_started":1}
I20260924 09:57:05.444051 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
/tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790243735634835-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790243735634835-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790243735634835-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790243735634835-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40515
--builtin_ntp_servers=127.0.97.20:39687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260924 09:57:05.444509 10402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:57:05.438840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40515 (local address 127.0.97.1:57037)
0924 09:57:05.439701 (+ 861us) negotiation.cc:107] Waiting for socket to connect
0924 09:57:05.439742 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 09:57:05.441221 (+ 1479us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:57:05.441872 (+ 651us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:57:05.441880 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:57:05.441971 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0924 09:57:05.442546 (+ 575us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 09:57:05.442548 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 09:57:05.442584 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 09:57:05.442666 (+ 82us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 09:57:05.443103 (+ 437us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 09:57:05.443111 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 09:57:05.443116 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 09:57:05.443367 (+ 251us) client_negotiation.cc:240] Negotiation successful
0924 09:57:05.443608 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":247,"threads_started":1}
I20260924 09:57:05.448343 10399 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40515
I20260924 09:57:05.448588 10399 heartbeater.cc:464] Registering TS with master...
I20260924 09:57:05.449105 10399 heartbeater.cc:511] Master 127.0.97.62:40515 requested a full tablet report, sending...
I20260924 09:57:05.450940 10213 ts_manager.cc:194] Registered new tserver with Master: e270115e259c4a93a12a0ae52520026e (127.0.97.1:40071)
W20260924 09:57:05.756541 10403 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 09:57:05.756812 10403 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 09:57:05.756894 10403 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 09:57:05.763846 10403 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 09:57:05.763939 10403 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 09:57:05.764050 10403 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260924 09:57:05.773949 10403 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790243735634835-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790243735634835-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790243735634835-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40515
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790243735634835-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 09:57:05.775141 10403 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 09:57:05.776870 10403 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 09:57:05.789124 10409 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:57:05.789191 10408 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:57:05.789626 10411 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 09:57:05.790051 10403 server_base.cc:1034] running on GCE node
I20260924 09:57:05.790627 10403 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 09:57:05.791569 10403 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 09:57:05.792762 10403 hybrid_clock.cc:648] HybridClock initialized: now 1790243825792713 us; error 41 us; skew 500 ppm
I20260924 09:57:05.793068 10403 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 09:57:05.795130 10403 webserver.cc:460] Webserver started at http://127.0.97.2:38663/ using document root <none> and password file <none>
I20260924 09:57:05.795802 10403 fs_manager.cc:359] Metadata directory not provided
I20260924 09:57:05.795897 10403 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 09:57:05.796193 10403 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 09:57:05.798935 10403 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790243735634835-388-0/minicluster-data/ts-1/data/instance:
uuid: "bd6833f21d064f73b53078e3b1b315c0"
format_stamp: "Formatted at 2026-09-24 09:57:05 on dist-test-slave-w8d2"
server_key: "b38498a494eacc97d28c893b10d7e801"
server_key_iv: "62b05ceca49a8cab4ad7077a81a5f49d"
server_key_version: "encryptionkey@0"
I20260924 09:57:05.799687 10403 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790243735634835-388-0/minicluster-data/ts-1/wal/instance:
uuid: "bd6833f21d064f73b53078e3b1b315c0"
format_stamp: "Formatted at 2026-09-24 09:57:05 on dist-test-slave-w8d2"
server_key: "b38498a494eacc97d28c893b10d7e801"
server_key_iv: "62b05ceca49a8cab4ad7077a81a5f49d"
server_key_version: "encryptionkey@0"
I20260924 09:57:05.804497 10403 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.002s
I20260924 09:57:05.807660 10417 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 09:57:05.808874 10403 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260924 09:57:05.809019 10403 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790243735634835-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790243735634835-388-0/minicluster-data/ts-1/wal
uuid: "bd6833f21d064f73b53078e3b1b315c0"
format_stamp: "Formatted at 2026-09-24 09:57:05 on dist-test-slave-w8d2"
server_key: "b38498a494eacc97d28c893b10d7e801"
server_key_iv: "62b05ceca49a8cab4ad7077a81a5f49d"
server_key_version: "encryptionkey@0"
I20260924 09:57:05.809224 10403 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790243735634835-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790243735634835-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790243735634835-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 09:57:05.814504 10403 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 09:57:05.815292 10403 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 09:57:05.815528 10403 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 09:57:05.816352 10403 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 09:57:05.817620 10403 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 09:57:05.817710 10403 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 09:57:05.817817 10403 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 09:57:05.817886 10403 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 09:57:05.846217 10403 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:39659
I20260924 09:57:05.846256 10529 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:39659 every 8 connection(s)
I20260924 09:57:05.847514 10403 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790243735634835-388-0/minicluster-data/ts-1/data/info.pb
I20260924 09:57:05.854905 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu as pid 10403
I20260924 09:57:05.854998 10401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:57:05.849844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:39619 (local address 127.0.97.62:40515)
0924 09:57:05.849994 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0924 09:57:05.849998 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 09:57:05.851407 (+ 1409us) server_negotiation.cc:407] Connection header received
0924 09:57:05.853021 (+ 1614us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:57:05.853027 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:57:05.853088 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:57:05.853172 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:57:05.854347 (+ 1175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:57:05.854352 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:57:05.854355 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 09:57:05.854370 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:57:05.854399 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 09:57:05.854410 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:57:05.854516 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0924 09:57:05.854680 (+ 164us) server_negotiation.cc:299] Negotiation successful
0924 09:57:05.854742 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 09:57:05.855051 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790243735634835-388-0/minicluster-data/ts-1/wal/instance
I20260924 09:57:05.855482 388 external_mini_cluster.cc:1428] Setting key 99aeb28ebec0e6bdf8a6a3113afdc22b
I20260924 09:57:05.855940 10532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:57:05.850356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40515 (local address 127.0.97.2:39619)
0924 09:57:05.851252 (+ 896us) negotiation.cc:107] Waiting for socket to connect
0924 09:57:05.851297 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 09:57:05.852752 (+ 1455us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:57:05.853399 (+ 647us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:57:05.853407 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:57:05.853497 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0924 09:57:05.854060 (+ 563us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 09:57:05.854062 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 09:57:05.854096 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 09:57:05.854177 (+ 81us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 09:57:05.854492 (+ 315us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 09:57:05.854502 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 09:57:05.854510 (+ 8us) client_negotiation.cc:769] Sending connection context
0924 09:57:05.854746 (+ 236us) client_negotiation.cc:240] Negotiation successful
0924 09:57:05.854994 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":267,"threads_started":1}
I20260924 09:57:05.857204 10530 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40515
I20260924 09:57:05.857458 10530 heartbeater.cc:464] Registering TS with master...
I20260924 09:57:05.857968 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
/tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790243735634835-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790243735634835-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790243735634835-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790243735634835-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40515
--builtin_ntp_servers=127.0.97.20:39687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260924 09:57:05.857980 10530 heartbeater.cc:511] Master 127.0.97.62:40515 requested a full tablet report, sending...
I20260924 09:57:05.859252 10213 ts_manager.cc:194] Registered new tserver with Master: bd6833f21d064f73b53078e3b1b315c0 (127.0.97.2:39659)
W20260924 09:57:06.162595 10533 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 09:57:06.162873 10533 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 09:57:06.162997 10533 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 09:57:06.170210 10533 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 09:57:06.170305 10533 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 09:57:06.170442 10533 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260924 09:57:06.179227 10533 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790243735634835-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790243735634835-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790243735634835-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40515
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790243735634835-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 09:57:06.180452 10533 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 09:57:06.182078 10533 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 09:57:06.193926 10538 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:57:06.193894 10539 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:57:06.193960 10541 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 09:57:06.194913 10533 server_base.cc:1034] running on GCE node
I20260924 09:57:06.195494 10533 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 09:57:06.196357 10533 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 09:57:06.197539 10533 hybrid_clock.cc:648] HybridClock initialized: now 1790243826197488 us; error 44 us; skew 500 ppm
I20260924 09:57:06.197813 10533 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 09:57:06.199831 10533 webserver.cc:460] Webserver started at http://127.0.97.3:39983/ using document root <none> and password file <none>
I20260924 09:57:06.200467 10533 fs_manager.cc:359] Metadata directory not provided
I20260924 09:57:06.200547 10533 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 09:57:06.200834 10533 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 09:57:06.203660 10533 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790243735634835-388-0/minicluster-data/ts-2/data/instance:
uuid: "bcc98aefb07d4014819c61f7011f00cd"
format_stamp: "Formatted at 2026-09-24 09:57:06 on dist-test-slave-w8d2"
server_key: "7d1e8a49ecbce4ae2a89eda6ce604af2"
server_key_iv: "efdff399487536dd667d0f2874e5bd7b"
server_key_version: "encryptionkey@0"
I20260924 09:57:06.204377 10533 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790243735634835-388-0/minicluster-data/ts-2/wal/instance:
uuid: "bcc98aefb07d4014819c61f7011f00cd"
format_stamp: "Formatted at 2026-09-24 09:57:06 on dist-test-slave-w8d2"
server_key: "7d1e8a49ecbce4ae2a89eda6ce604af2"
server_key_iv: "efdff399487536dd667d0f2874e5bd7b"
server_key_version: "encryptionkey@0"
I20260924 09:57:06.208981 10533 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 09:57:06.212085 10547 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 09:57:06.213279 10533 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260924 09:57:06.213414 10533 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790243735634835-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790243735634835-388-0/minicluster-data/ts-2/wal
uuid: "bcc98aefb07d4014819c61f7011f00cd"
format_stamp: "Formatted at 2026-09-24 09:57:06 on dist-test-slave-w8d2"
server_key: "7d1e8a49ecbce4ae2a89eda6ce604af2"
server_key_iv: "efdff399487536dd667d0f2874e5bd7b"
server_key_version: "encryptionkey@0"
I20260924 09:57:06.213601 10533 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790243735634835-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790243735634835-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790243735634835-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 09:57:06.218822 10533 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 09:57:06.219578 10533 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 09:57:06.219810 10533 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 09:57:06.220618 10533 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 09:57:06.221884 10533 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 09:57:06.221985 10533 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 09:57:06.222084 10533 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 09:57:06.222157 10533 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 09:57:06.250599 10533 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:35435
I20260924 09:57:06.250670 10659 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:35435 every 8 connection(s)
I20260924 09:57:06.252002 10533 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790243735634835-388-0/minicluster-data/ts-2/data/info.pb
I20260924 09:57:06.257041 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu as pid 10533
I20260924 09:57:06.257169 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790243735634835-388-0/minicluster-data/ts-2/wal/instance
I20260924 09:57:06.257586 388 external_mini_cluster.cc:1428] Setting key 5734a063c696ce8400a3c78ce44a60d8
I20260924 09:57:06.259601 10401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:57:06.254310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:53523 (local address 127.0.97.62:40515)
0924 09:57:06.254512 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0924 09:57:06.254519 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 09:57:06.255856 (+ 1337us) server_negotiation.cc:407] Connection header received
0924 09:57:06.257433 (+ 1577us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:57:06.257438 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:57:06.257502 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:57:06.257589 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:57:06.258930 (+ 1341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:57:06.258937 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:57:06.258941 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 09:57:06.258960 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:57:06.258989 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 09:57:06.259000 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:57:06.259099 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0924 09:57:06.259291 (+ 192us) server_negotiation.cc:299] Negotiation successful
0924 09:57:06.259352 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260924 09:57:06.260615 10662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:57:06.254835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40515 (local address 127.0.97.3:53523)
0924 09:57:06.255710 (+ 875us) negotiation.cc:107] Waiting for socket to connect
0924 09:57:06.255748 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 09:57:06.257161 (+ 1413us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:57:06.257818 (+ 657us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:57:06.257827 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:57:06.257919 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0924 09:57:06.258646 (+ 727us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 09:57:06.258649 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 09:57:06.258685 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 09:57:06.258769 (+ 84us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 09:57:06.259084 (+ 315us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 09:57:06.259094 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 09:57:06.259101 (+ 7us) client_negotiation.cc:769] Sending connection context
0924 09:57:06.259366 (+ 265us) client_negotiation.cc:240] Negotiation successful
0924 09:57:06.259607 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":257,"threads_started":1}
I20260924 09:57:06.261734 10660 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40515
I20260924 09:57:06.261971 10660 heartbeater.cc:464] Registering TS with master...
I20260924 09:57:06.262480 10660 heartbeater.cc:511] Master 127.0.97.62:40515 requested a full tablet report, sending...
I20260924 09:57:06.263672 10213 ts_manager.cc:194] Registered new tserver with Master: bcc98aefb07d4014819c61f7011f00cd (127.0.97.3:35435)
I20260924 09:57:06.272105 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260924 09:57:06.277629 10401 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 09:57:06.276240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42970 (local address 127.0.97.62:40515)
0924 09:57:06.276392 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0924 09:57:06.276396 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 09:57:06.276865 (+ 469us) server_negotiation.cc:407] Connection header received
0924 09:57:06.277011 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:57:06.277015 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:57:06.277064 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:57:06.277141 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:57:06.277444 (+ 303us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42970: BlockingRecv error: recv got EOF from 127.0.0.1:42970 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 09:57:06.278398 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu pid 10272
I20260924 09:57:06.281342 10669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:57:06.279216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58894 (local address 127.0.97.1:40071)
0924 09:57:06.279693 (+ 477us) server_negotiation.cc:206] Beginning negotiation
0924 09:57:06.279699 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 09:57:06.279723 (+ 24us) server_negotiation.cc:407] Connection header received
0924 09:57:06.279773 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:57:06.279779 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:57:06.279980 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:57:06.280091 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:57:06.280804 (+ 713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:57:06.280810 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:57:06.280821 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 09:57:06.280836 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:57:06.280876 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 09:57:06.280890 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:57:06.280982 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0924 09:57:06.281049 (+ 67us) server_negotiation.cc:299] Negotiation successful
0924 09:57:06.281120 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343,"thread_start_us":206,"threads_started":1}
I20260924 09:57:06.282444 10394 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 09:57:06.363035 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu with pid 10272
I20260924 09:57:06.375960 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu pid 10403
I20260924 09:57:06.379189 10671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:57:06.376866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40996 (local address 127.0.97.2:39659)
0924 09:57:06.377373 (+ 507us) server_negotiation.cc:206] Beginning negotiation
0924 09:57:06.377380 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 09:57:06.377406 (+ 26us) server_negotiation.cc:407] Connection header received
0924 09:57:06.377465 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:57:06.377471 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:57:06.377686 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:57:06.377807 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:57:06.378614 (+ 807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:57:06.378620 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:57:06.378632 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 09:57:06.378649 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:57:06.378690 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 09:57:06.378704 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:57:06.378803 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0924 09:57:06.378886 (+ 83us) server_negotiation.cc:299] Negotiation successful
0924 09:57:06.378964 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":376,"thread_start_us":259,"threads_started":1}
I20260924 09:57:06.380371 10525 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 09:57:06.458688 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu with pid 10403
I20260924 09:57:06.470872 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu pid 10533
I20260924 09:57:06.474372 10673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:57:06.471649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41930 (local address 127.0.97.3:35435)
0924 09:57:06.472182 (+ 533us) server_negotiation.cc:206] Beginning negotiation
0924 09:57:06.472189 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 09:57:06.472216 (+ 27us) server_negotiation.cc:407] Connection header received
0924 09:57:06.472285 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:57:06.472294 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:57:06.472503 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:57:06.472643 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:57:06.473682 (+ 1039us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:57:06.473688 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:57:06.473700 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 09:57:06.473711 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:57:06.473749 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 09:57:06.473767 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:57:06.473914 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0924 09:57:06.473983 (+ 69us) server_negotiation.cc:299] Negotiation successful
0924 09:57:06.474090 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":395,"thread_start_us":243,"threads_started":1}
I20260924 09:57:06.475557 10655 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 09:57:06.554520 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu with pid 10533
I20260924 09:57:06.566700 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu pid 10182
I20260924 09:57:06.567871 10243 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 09:57:06.664553 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu with pid 10182
2026-09-24T09:57:06Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2227 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790243735634835-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 09:57:06 dist-test-slave-w8d2 krb5kdc[10678](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 09:57:06 dist-test-slave-w8d2 krb5kdc[10678](info): set up 2 sockets
Sep 24 09:57:06 dist-test-slave-w8d2 krb5kdc[10678](info): commencing operation
krb5kdc: starting...
W20260924 09:57:07.667943 388 mini_kdc.cc:121] Time spent starting KDC: real 0.968s user 0.007s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790243735634835-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790243735634835-388-0/krb5kdc/test-user.keytab.
Sep 24 09:57:07 dist-test-slave-w8d2 krb5kdc[10678](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243827, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T09:57:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T09:57:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790243735634835-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790243735634835-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260924 09:57:07.882097 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
/tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790243735634835-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790243735634835-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790243735634835-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790243735634835-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42477
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790243735634835-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:42477
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790243735634835-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790243735634835-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790243735634835-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 09:57:08.191399 10691 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 09:57:08.191689 10691 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 09:57:08.191828 10691 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 09:57:08.198948 10691 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 09:57:08.199066 10691 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 09:57:08.199141 10691 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 09:57:08.199215 10691 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 09:57:08.199301 10691 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 09:57:08.209079 10691 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790243735634835-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790243735634835-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:42477
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790243735634835-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42477
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790243735634835-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790243735634835-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 09:57:08.210457 10691 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 09:57:08.211843 10691 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 09:57:08.223199 10699 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:57:08.223304 10696 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:57:08.223277 10697 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 09:57:08.224064 10691 server_base.cc:1034] running on GCE node
I20260924 09:57:08.224789 10691 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 09:57:08.226177 10691 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 09:57:08.227336 10691 hybrid_clock.cc:648] HybridClock initialized: now 1790243828227283 us; error 32 us; skew 500 ppm
I20260924 09:57:08.227674 10691 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 09:57:08 dist-test-slave-w8d2 krb5kdc[10678](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243828, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 09:57:08.230558 10691 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260924 09:57:08.232041 10691 webserver.cc:460] Webserver started at http://127.0.97.62:34227/ using document root <none> and password file <none>
I20260924 09:57:08.232671 10691 fs_manager.cc:359] Metadata directory not provided
I20260924 09:57:08.232769 10691 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 09:57:08.233024 10691 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 09:57:08.235895 10691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790243735634835-388-0/minicluster-data/master-0/data/instance:
uuid: "340849d406bd452b952004a0ae6c375b"
format_stamp: "Formatted at 2026-09-24 09:57:08 on dist-test-slave-w8d2"
server_key: "f4ea8c78c42eaf15598d8f0c41ad0667"
server_key_iv: "bf95b56cb3eb1a5733526dd5f1057a03"
server_key_version: "encryptionkey@0"
I20260924 09:57:08.236598 10691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790243735634835-388-0/minicluster-data/master-0/wal/instance:
uuid: "340849d406bd452b952004a0ae6c375b"
format_stamp: "Formatted at 2026-09-24 09:57:08 on dist-test-slave-w8d2"
server_key: "f4ea8c78c42eaf15598d8f0c41ad0667"
server_key_iv: "bf95b56cb3eb1a5733526dd5f1057a03"
server_key_version: "encryptionkey@0"
I20260924 09:57:08.241504 10691 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260924 09:57:08.244845 10706 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 09:57:08.246094 10691 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 09:57:08.246227 10691 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790243735634835-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790243735634835-388-0/minicluster-data/master-0/wal
uuid: "340849d406bd452b952004a0ae6c375b"
format_stamp: "Formatted at 2026-09-24 09:57:08 on dist-test-slave-w8d2"
server_key: "f4ea8c78c42eaf15598d8f0c41ad0667"
server_key_iv: "bf95b56cb3eb1a5733526dd5f1057a03"
server_key_version: "encryptionkey@0"
I20260924 09:57:08.246382 10691 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790243735634835-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790243735634835-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790243735634835-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 09:57:08.268986 10691 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 09:57:08.272763 10691 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 09:57:08.273002 10691 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 09:57:08.289698 10691 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:42477
I20260924 09:57:08.289750 10757 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:42477 every 8 connection(s)
I20260924 09:57:08.291023 10691 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790243735634835-388-0/minicluster-data/master-0/data/info.pb
I20260924 09:57:08.294458 10758 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 09:57:08.300236 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu as pid 10691
I20260924 09:57:08.300467 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790243735634835-388-0/minicluster-data/master-0/wal/instance
I20260924 09:57:08.301214 388 external_mini_cluster.cc:1428] Setting key dec0a652ee04853f73a7a5266b872c4d
I20260924 09:57:08.305964 10758 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 340849d406bd452b952004a0ae6c375b: Bootstrap starting.
I20260924 09:57:08.309310 10758 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 340849d406bd452b952004a0ae6c375b: Neither blocks nor log segments found. Creating new log.
I20260924 09:57:08.310344 10758 log.cc:824] T 00000000000000000000000000000000 P 340849d406bd452b952004a0ae6c375b: Log is configured to *not* fsync() on all Append() calls
I20260924 09:57:08.312961 10758 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 340849d406bd452b952004a0ae6c375b: No bootstrap required, opened a new log
Sep 24 09:57:08 dist-test-slave-w8d2 krb5kdc[10678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243827, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 09:57:08.321049 10758 raft_consensus.cc:348] T 00000000000000000000000000000000 P 340849d406bd452b952004a0ae6c375b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "340849d406bd452b952004a0ae6c375b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42477 } }
I20260924 09:57:08.321427 10758 raft_consensus.cc:374] T 00000000000000000000000000000000 P 340849d406bd452b952004a0ae6c375b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 09:57:08.321527 10758 raft_consensus.cc:729] T 00000000000000000000000000000000 P 340849d406bd452b952004a0ae6c375b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 340849d406bd452b952004a0ae6c375b, State: Initialized, Role: FOLLOWER
I20260924 09:57:08.322141 10758 consensus_queue.cc:260] T 00000000000000000000000000000000 P 340849d406bd452b952004a0ae6c375b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "340849d406bd452b952004a0ae6c375b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42477 } }
I20260924 09:57:08.322391 10758 raft_consensus.cc:388] T 00000000000000000000000000000000 P 340849d406bd452b952004a0ae6c375b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 09:57:08.322516 10758 raft_consensus.cc:482] T 00000000000000000000000000000000 P 340849d406bd452b952004a0ae6c375b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 09:57:08.322672 10758 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 340849d406bd452b952004a0ae6c375b [term 0 FOLLOWER]: Advancing to term 1
I20260924 09:57:08.323655 10761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:57:08.303392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54402 (local address 127.0.97.62:42477)
0924 09:57:08.304184 (+ 792us) server_negotiation.cc:206] Beginning negotiation
0924 09:57:08.304199 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 09:57:08.304246 (+ 47us) server_negotiation.cc:407] Connection header received
0924 09:57:08.306576 (+ 2330us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:57:08.306605 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:57:08.307122 (+ 517us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:57:08.307587 (+ 465us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:57:08.308948 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:57:08.310365 (+ 1417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:57:08.311648 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:57:08.312323 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:57:08.312479 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:57:08.316369 (+ 3890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:57:08.316377 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:57:08.316391 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:57:08.316433 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:57:08.319476 (+ 3043us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:57:08.320250 (+ 774us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:57:08.320257 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:57:08.320263 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:57:08.320432 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:57:08.320813 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:57:08.320816 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:57:08.320819 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:57:08.321328 (+ 509us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:57:08.321608 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0924 09:57:08.322086 (+ 478us) server_negotiation.cc:299] Negotiation successful
0924 09:57:08.322526 (+ 440us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":546,"thread_start_us":335,"threads_started":1}
I20260924 09:57:08.325383 10758 raft_consensus.cc:504] T 00000000000000000000000000000000 P 340849d406bd452b952004a0ae6c375b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "340849d406bd452b952004a0ae6c375b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42477 } }
I20260924 09:57:08.325820 10758 leader_election.cc:302] T 00000000000000000000000000000000 P 340849d406bd452b952004a0ae6c375b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 340849d406bd452b952004a0ae6c375b; no voters:
I20260924 09:57:08.326421 10758 leader_election.cc:288] T 00000000000000000000000000000000 P 340849d406bd452b952004a0ae6c375b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 09:57:08.326550 10763 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 340849d406bd452b952004a0ae6c375b [term 1 FOLLOWER]: Leader election won for term 1
I20260924 09:57:08.327137 10763 raft_consensus.cc:686] T 00000000000000000000000000000000 P 340849d406bd452b952004a0ae6c375b [term 1 LEADER]: Becoming Leader. State: Replica: 340849d406bd452b952004a0ae6c375b, State: Running, Role: LEADER
I20260924 09:57:08.327740 10763 consensus_queue.cc:237] T 00000000000000000000000000000000 P 340849d406bd452b952004a0ae6c375b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "340849d406bd452b952004a0ae6c375b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42477 } }
I20260924 09:57:08.328271 10758 sys_catalog.cc:564] T 00000000000000000000000000000000 P 340849d406bd452b952004a0ae6c375b [sys.catalog]: configured and running, proceeding with master startup.
I20260924 09:57:08.330988 10764 sys_catalog.cc:455] T 00000000000000000000000000000000 P 340849d406bd452b952004a0ae6c375b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "340849d406bd452b952004a0ae6c375b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "340849d406bd452b952004a0ae6c375b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42477 } } }
I20260924 09:57:08.330963 10765 sys_catalog.cc:455] T 00000000000000000000000000000000 P 340849d406bd452b952004a0ae6c375b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 340849d406bd452b952004a0ae6c375b. Latest consensus state: current_term: 1 leader_uuid: "340849d406bd452b952004a0ae6c375b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "340849d406bd452b952004a0ae6c375b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42477 } } }
I20260924 09:57:08.331378 10764 sys_catalog.cc:458] T 00000000000000000000000000000000 P 340849d406bd452b952004a0ae6c375b [sys.catalog]: This master's current role is: LEADER
I20260924 09:57:08.331389 10765 sys_catalog.cc:458] T 00000000000000000000000000000000 P 340849d406bd452b952004a0ae6c375b [sys.catalog]: This master's current role is: LEADER
I20260924 09:57:08.338994 10771 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 09:57:08.343406 10771 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 09:57:08.351372 10771 catalog_manager.cc:1347] Generated new cluster ID: 1c830ee02bcf45f4a5532e181d40a7ac
I20260924 09:57:08.351482 10771 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 09:57:08.365733 10771 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 09:57:08.367578 10771 catalog_manager.cc:1504] Loading token signing keys...
I20260924 09:57:08.382004 10771 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 340849d406bd452b952004a0ae6c375b: Generated new TSK 0
I20260924 09:57:08.383105 10771 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790243735634835-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790243735634835-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260924 09:57:08.434553 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
/tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790243735634835-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790243735634835-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790243735634835-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790243735634835-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42477
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790243735634835-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790243735634835-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790243735634835-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790243735634835-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 09:57:08.746778 10784 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 09:57:08.747027 10784 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 09:57:08.747166 10784 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 09:57:08.755148 10784 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 09:57:08.755273 10784 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 09:57:08.755429 10784 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260924 09:57:08.764861 10784 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790243735634835-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790243735634835-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790243735634835-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790243735634835-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42477
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790243735634835-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 09:57:08.766089 10784 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 09:57:08.767333 10784 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 09:57:08.779093 10789 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:57:08.779105 10790 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:57:08.779167 10792 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 09:57:08.779891 10784 server_base.cc:1034] running on GCE node
I20260924 09:57:08.780458 10784 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 09:57:08.781325 10784 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 09:57:08.782511 10784 hybrid_clock.cc:648] HybridClock initialized: now 1790243828782456 us; error 52 us; skew 500 ppm
I20260924 09:57:08.782860 10784 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 09:57:08 dist-test-slave-w8d2 krb5kdc[10678](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243828, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 09:57:08.785823 10784 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260924 09:57:08.787128 10784 webserver.cc:460] Webserver started at http://127.0.97.1:36219/ using document root <none> and password file <none>
I20260924 09:57:08.787763 10784 fs_manager.cc:359] Metadata directory not provided
I20260924 09:57:08.787860 10784 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 09:57:08.788116 10784 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 09:57:08.790735 10784 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790243735634835-388-0/minicluster-data/ts-0/data/instance:
uuid: "22c41fefcd404cb387906261d16c8f42"
format_stamp: "Formatted at 2026-09-24 09:57:08 on dist-test-slave-w8d2"
server_key: "33882a9327c6b34dcc78213d7d65b0fa"
server_key_iv: "6bbc4a948b6c6375195cb26ff927ced9"
server_key_version: "encryptionkey@0"
I20260924 09:57:08.791462 10784 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790243735634835-388-0/minicluster-data/ts-0/wal/instance:
uuid: "22c41fefcd404cb387906261d16c8f42"
format_stamp: "Formatted at 2026-09-24 09:57:08 on dist-test-slave-w8d2"
server_key: "33882a9327c6b34dcc78213d7d65b0fa"
server_key_iv: "6bbc4a948b6c6375195cb26ff927ced9"
server_key_version: "encryptionkey@0"
I20260924 09:57:08.796015 10784 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 09:57:08.799078 10799 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 09:57:08.800355 10784 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260924 09:57:08.800477 10784 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790243735634835-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790243735634835-388-0/minicluster-data/ts-0/wal
uuid: "22c41fefcd404cb387906261d16c8f42"
format_stamp: "Formatted at 2026-09-24 09:57:08 on dist-test-slave-w8d2"
server_key: "33882a9327c6b34dcc78213d7d65b0fa"
server_key_iv: "6bbc4a948b6c6375195cb26ff927ced9"
server_key_version: "encryptionkey@0"
I20260924 09:57:08.800652 10784 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790243735634835-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790243735634835-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790243735634835-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 09:57:08.812598 10784 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 09:57:08.815982 10784 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 09:57:08.816221 10784 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 09:57:08.817008 10784 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 09:57:08.818156 10784 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 09:57:08.818221 10784 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 09:57:08.818326 10784 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 09:57:08.818373 10784 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 09:57:08.846460 10784 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:38181
I20260924 09:57:08.846493 10911 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:38181 every 8 connection(s)
I20260924 09:57:08.847635 10784 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790243735634835-388-0/minicluster-data/ts-0/data/info.pb
I20260924 09:57:08.854113 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu as pid 10784
I20260924 09:57:08.854313 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790243735634835-388-0/minicluster-data/ts-0/wal/instance
I20260924 09:57:08.854978 388 external_mini_cluster.cc:1428] Setting key 19a200b90dec9967e6520b17574f9ad0
Sep 24 09:57:08 dist-test-slave-w8d2 krb5kdc[10678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243828, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 09:57:08.866428 10914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:57:08.849827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:35893 (local address 127.0.97.62:42477)
0924 09:57:08.850212 (+ 385us) server_negotiation.cc:206] Beginning negotiation
0924 09:57:08.850217 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 09:57:08.851061 (+ 844us) server_negotiation.cc:407] Connection header received
0924 09:57:08.852788 (+ 1727us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:57:08.852794 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:57:08.852870 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:57:08.852961 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:57:08.854580 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:57:08.855383 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:57:08.857173 (+ 1790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:57:08.857530 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:57:08.857618 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:57:08.861245 (+ 3627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:57:08.861250 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:57:08.861253 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:57:08.861281 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:57:08.863601 (+ 2320us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:57:08.864379 (+ 778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:57:08.864382 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:57:08.864385 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:57:08.864484 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:57:08.864866 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:57:08.864870 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:57:08.864873 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:57:08.865190 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:57:08.865315 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0924 09:57:08.865883 (+ 568us) server_negotiation.cc:299] Negotiation successful
0924 09:57:08.866081 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":209,"threads_started":1}
I20260924 09:57:08.867208 10915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:57:08.850247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42477 (local address 127.0.97.1:35893)
0924 09:57:08.850903 (+ 656us) negotiation.cc:107] Waiting for socket to connect
0924 09:57:08.850953 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0924 09:57:08.852544 (+ 1591us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:57:08.853195 (+ 651us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:57:08.853204 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:57:08.853661 (+ 457us) client_negotiation.cc:189] Negotiated authn=SASL
0924 09:57:08.854416 (+ 755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:57:08.854440 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:57:08.855519 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:57:08.855524 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:57:08.857034 (+ 1510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:57:08.857047 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:57:08.857733 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:57:08.857740 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:57:08.858001 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:57:08.858644 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 09:57:08.858671 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0924 09:57:08.861040 (+ 2369us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 09:57:08.863726 (+ 2686us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 09:57:08.863734 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 09:57:08.863750 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 09:57:08.864285 (+ 535us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 09:57:08.864579 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 09:57:08.864584 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 09:57:08.864587 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 09:57:08.864770 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 09:57:08.865288 (+ 518us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 09:57:08.865293 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 09:57:08.865584 (+ 291us) client_negotiation.cc:769] Sending connection context
0924 09:57:08.865792 (+ 208us) client_negotiation.cc:240] Negotiation successful
0924 09:57:08.866159 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":255,"threads_started":1}
I20260924 09:57:08.868606 10912 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42477
I20260924 09:57:08.868898 10912 heartbeater.cc:464] Registering TS with master...
I20260924 09:57:08.869665 10912 heartbeater.cc:511] Master 127.0.97.62:42477 requested a full tablet report, sending...
I20260924 09:57:08.871738 10723 ts_manager.cc:194] Registered new tserver with Master: 22c41fefcd404cb387906261d16c8f42 (127.0.97.1:38181)
I20260924 09:57:08.873195 10723 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:35893
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790243735634835-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790243735634835-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260924 09:57:08.902427 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
/tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790243735634835-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790243735634835-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790243735634835-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790243735634835-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42477
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790243735634835-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790243735634835-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790243735634835-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790243735634835-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 09:57:09.199980 10918 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 09:57:09.200260 10918 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 09:57:09.200384 10918 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 09:57:09.207540 10918 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 09:57:09.207654 10918 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 09:57:09.207811 10918 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260924 09:57:09.217191 10918 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790243735634835-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790243735634835-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790243735634835-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790243735634835-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42477
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790243735634835-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 09:57:09.218364 10918 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 09:57:09.219627 10918 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 09:57:09.231586 10923 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:57:09.231535 10924 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:57:09.231623 10926 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 09:57:09.232482 10918 server_base.cc:1034] running on GCE node
I20260924 09:57:09.233037 10918 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 09:57:09.233915 10918 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 09:57:09.235100 10918 hybrid_clock.cc:648] HybridClock initialized: now 1790243829235048 us; error 50 us; skew 500 ppm
I20260924 09:57:09.235421 10918 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 09:57:09 dist-test-slave-w8d2 krb5kdc[10678](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243829, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 09:57:09.238298 10918 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260924 09:57:09.239661 10918 webserver.cc:460] Webserver started at http://127.0.97.2:40927/ using document root <none> and password file <none>
I20260924 09:57:09.240275 10918 fs_manager.cc:359] Metadata directory not provided
I20260924 09:57:09.240365 10918 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 09:57:09.240581 10918 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 09:57:09.243168 10918 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790243735634835-388-0/minicluster-data/ts-1/data/instance:
uuid: "ac74072ff0ca4272937a2603959a2dab"
format_stamp: "Formatted at 2026-09-24 09:57:09 on dist-test-slave-w8d2"
server_key: "b343a19d31b2a6551890974de652c689"
server_key_iv: "f19e6f33389d6d356b7179a1667f4f76"
server_key_version: "encryptionkey@0"
I20260924 09:57:09.243942 10918 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790243735634835-388-0/minicluster-data/ts-1/wal/instance:
uuid: "ac74072ff0ca4272937a2603959a2dab"
format_stamp: "Formatted at 2026-09-24 09:57:09 on dist-test-slave-w8d2"
server_key: "b343a19d31b2a6551890974de652c689"
server_key_iv: "f19e6f33389d6d356b7179a1667f4f76"
server_key_version: "encryptionkey@0"
I20260924 09:57:09.248471 10918 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 09:57:09.251511 10933 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 09:57:09.252687 10918 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 09:57:09.252803 10918 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790243735634835-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790243735634835-388-0/minicluster-data/ts-1/wal
uuid: "ac74072ff0ca4272937a2603959a2dab"
format_stamp: "Formatted at 2026-09-24 09:57:09 on dist-test-slave-w8d2"
server_key: "b343a19d31b2a6551890974de652c689"
server_key_iv: "f19e6f33389d6d356b7179a1667f4f76"
server_key_version: "encryptionkey@0"
I20260924 09:57:09.252951 10918 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790243735634835-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790243735634835-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790243735634835-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 09:57:09.273501 10918 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 09:57:09.277136 10918 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 09:57:09.277371 10918 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 09:57:09.278183 10918 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 09:57:09.279404 10918 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 09:57:09.279479 10918 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 09:57:09.279573 10918 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 09:57:09.279616 10918 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 09:57:09.313146 10918 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:38047
I20260924 09:57:09.313186 11045 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:38047 every 8 connection(s)
I20260924 09:57:09.314276 10918 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790243735634835-388-0/minicluster-data/ts-1/data/info.pb
I20260924 09:57:09.319584 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu as pid 10918
I20260924 09:57:09.319754 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790243735634835-388-0/minicluster-data/ts-1/wal/instance
I20260924 09:57:09.320277 388 external_mini_cluster.cc:1428] Setting key 99698bb71b988c7f32babd67cc78eca3
Sep 24 09:57:09 dist-test-slave-w8d2 krb5kdc[10678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243829, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 09:57:09.334084 10914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:57:09.316544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:34797 (local address 127.0.97.62:42477)
0924 09:57:09.316710 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0924 09:57:09.316715 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 09:57:09.317930 (+ 1215us) server_negotiation.cc:407] Connection header received
0924 09:57:09.319876 (+ 1946us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:57:09.319881 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:57:09.319947 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:57:09.320034 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:57:09.321756 (+ 1722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:57:09.322536 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:57:09.324622 (+ 2086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:57:09.325014 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:57:09.325110 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:57:09.328668 (+ 3558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:57:09.328673 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:57:09.328676 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:57:09.328708 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:57:09.331344 (+ 2636us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:57:09.332017 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:57:09.332021 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:57:09.332023 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:57:09.332145 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:57:09.332537 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:57:09.332541 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:57:09.332543 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:57:09.332775 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:57:09.332896 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0924 09:57:09.333555 (+ 659us) server_negotiation.cc:299] Negotiation successful
0924 09:57:09.333769 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 09:57:09.335031 11048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:57:09.316993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42477 (local address 127.0.97.2:34797)
0924 09:57:09.317657 (+ 664us) negotiation.cc:107] Waiting for socket to connect
0924 09:57:09.317696 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 09:57:09.319584 (+ 1888us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:57:09.320306 (+ 722us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:57:09.320317 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:57:09.320957 (+ 640us) client_negotiation.cc:189] Negotiated authn=SASL
0924 09:57:09.321593 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:57:09.321609 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:57:09.322690 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:57:09.322695 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:57:09.324502 (+ 1807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:57:09.324515 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:57:09.325225 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:57:09.325230 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:57:09.325473 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:57:09.326108 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 09:57:09.326137 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0924 09:57:09.328472 (+ 2335us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 09:57:09.331472 (+ 3000us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 09:57:09.331481 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 09:57:09.331498 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 09:57:09.331892 (+ 394us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 09:57:09.332246 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 09:57:09.332250 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 09:57:09.332254 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 09:57:09.332426 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 09:57:09.332892 (+ 466us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 09:57:09.332899 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 09:57:09.333216 (+ 317us) client_negotiation.cc:769] Sending connection context
0924 09:57:09.333445 (+ 229us) client_negotiation.cc:240] Negotiation successful
0924 09:57:09.333952 (+ 507us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":248,"threads_started":1}
I20260924 09:57:09.336661 11046 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42477
I20260924 09:57:09.336956 11046 heartbeater.cc:464] Registering TS with master...
I20260924 09:57:09.337703 11046 heartbeater.cc:511] Master 127.0.97.62:42477 requested a full tablet report, sending...
I20260924 09:57:09.339190 10723 ts_manager.cc:194] Registered new tserver with Master: ac74072ff0ca4272937a2603959a2dab (127.0.97.2:38047)
I20260924 09:57:09.339969 10723 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:34797
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790243735634835-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790243735634835-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260924 09:57:09.367813 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
/tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790243735634835-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790243735634835-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790243735634835-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790243735634835-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42477
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790243735634835-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790243735634835-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790243735634835-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790243735634835-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 09:57:09.674436 11051 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 09:57:09.674688 11051 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 09:57:09.674808 11051 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 09:57:09.681794 11051 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 09:57:09.681896 11051 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 09:57:09.682049 11051 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260924 09:57:09.691357 11051 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790243735634835-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790243735634835-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790243735634835-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790243735634835-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42477
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790243735634835-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 09:57:09.692528 11051 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 09:57:09.693778 11051 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 09:57:09.705785 11056 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:57:09.705826 11059 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:57:09.705856 11057 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 09:57:09.706560 11051 server_base.cc:1034] running on GCE node
I20260924 09:57:09.707077 11051 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 09:57:09.707938 11051 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 09:57:09.709117 11051 hybrid_clock.cc:648] HybridClock initialized: now 1790243829709068 us; error 43 us; skew 500 ppm
I20260924 09:57:09.709383 11051 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 09:57:09 dist-test-slave-w8d2 krb5kdc[10678](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243829, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 09:57:09.712213 11051 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260924 09:57:09.713452 11051 webserver.cc:460] Webserver started at http://127.0.97.3:35293/ using document root <none> and password file <none>
I20260924 09:57:09.714040 11051 fs_manager.cc:359] Metadata directory not provided
I20260924 09:57:09.714128 11051 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 09:57:09.714349 11051 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 09:57:09.716993 11051 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790243735634835-388-0/minicluster-data/ts-2/data/instance:
uuid: "c74150c79b9747bdb31c6c44093f58af"
format_stamp: "Formatted at 2026-09-24 09:57:09 on dist-test-slave-w8d2"
server_key: "8cb8883df6460088eaaeec0f095a055f"
server_key_iv: "fb2f8cb0f8599eebf74770ce0854cceb"
server_key_version: "encryptionkey@0"
I20260924 09:57:09.717620 11051 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790243735634835-388-0/minicluster-data/ts-2/wal/instance:
uuid: "c74150c79b9747bdb31c6c44093f58af"
format_stamp: "Formatted at 2026-09-24 09:57:09 on dist-test-slave-w8d2"
server_key: "8cb8883df6460088eaaeec0f095a055f"
server_key_iv: "fb2f8cb0f8599eebf74770ce0854cceb"
server_key_version: "encryptionkey@0"
I20260924 09:57:09.722078 11051 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 09:57:09.725126 11066 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 09:57:09.726308 11051 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 09:57:09.726437 11051 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790243735634835-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790243735634835-388-0/minicluster-data/ts-2/wal
uuid: "c74150c79b9747bdb31c6c44093f58af"
format_stamp: "Formatted at 2026-09-24 09:57:09 on dist-test-slave-w8d2"
server_key: "8cb8883df6460088eaaeec0f095a055f"
server_key_iv: "fb2f8cb0f8599eebf74770ce0854cceb"
server_key_version: "encryptionkey@0"
I20260924 09:57:09.726593 11051 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790243735634835-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790243735634835-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790243735634835-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 09:57:09.744212 11051 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 09:57:09.747771 11051 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 09:57:09.748027 11051 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 09:57:09.748850 11051 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 09:57:09.750123 11051 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 09:57:09.750187 11051 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 09:57:09.750276 11051 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 09:57:09.750316 11051 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 09:57:09.778834 11051 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:39847
I20260924 09:57:09.778869 11178 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:39847 every 8 connection(s)
I20260924 09:57:09.780011 11051 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790243735634835-388-0/minicluster-data/ts-2/data/info.pb
I20260924 09:57:09.786643 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu as pid 11051
I20260924 09:57:09.786774 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790243735634835-388-0/minicluster-data/ts-2/wal/instance
I20260924 09:57:09.787178 388 external_mini_cluster.cc:1428] Setting key a692a217dc6c2aa2c084c62523702f75
Sep 24 09:57:09 dist-test-slave-w8d2 krb5kdc[10678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243829, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 09:57:09.797711 10914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:57:09.782311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:54767 (local address 127.0.97.62:42477)
0924 09:57:09.782470 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0924 09:57:09.782475 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 09:57:09.783654 (+ 1179us) server_negotiation.cc:407] Connection header received
0924 09:57:09.785420 (+ 1766us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:57:09.785426 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:57:09.785501 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:57:09.785595 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:57:09.787053 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:57:09.787851 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:57:09.789720 (+ 1869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:57:09.790199 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:57:09.790289 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:57:09.793332 (+ 3043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:57:09.793337 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:57:09.793340 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:57:09.793371 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:57:09.795484 (+ 2113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:57:09.796072 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:57:09.796076 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:57:09.796078 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:57:09.796174 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:57:09.796494 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:57:09.796497 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:57:09.796499 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:57:09.796726 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:57:09.796835 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0924 09:57:09.797346 (+ 511us) server_negotiation.cc:299] Negotiation successful
0924 09:57:09.797486 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 09:57:09.798602 11181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:57:09.782763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42477 (local address 127.0.97.3:54767)
0924 09:57:09.783489 (+ 726us) negotiation.cc:107] Waiting for socket to connect
0924 09:57:09.783541 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0924 09:57:09.785158 (+ 1617us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:57:09.785832 (+ 674us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:57:09.785840 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:57:09.786312 (+ 472us) client_negotiation.cc:189] Negotiated authn=SASL
0924 09:57:09.786879 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:57:09.786922 (+ 43us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:57:09.787980 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:57:09.787984 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:57:09.789579 (+ 1595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:57:09.789592 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:57:09.790374 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:57:09.790379 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:57:09.790559 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:57:09.791115 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 09:57:09.791138 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 09:57:09.793156 (+ 2018us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 09:57:09.795591 (+ 2435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 09:57:09.795597 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 09:57:09.795607 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0924 09:57:09.795989 (+ 382us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 09:57:09.796258 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 09:57:09.796262 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 09:57:09.796264 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 09:57:09.796420 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 09:57:09.796821 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 09:57:09.796825 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 09:57:09.797081 (+ 256us) client_negotiation.cc:769] Sending connection context
0924 09:57:09.797286 (+ 205us) client_negotiation.cc:240] Negotiation successful
0924 09:57:09.797609 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":276,"threads_started":1}
I20260924 09:57:09.799832 11179 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42477
I20260924 09:57:09.800082 11179 heartbeater.cc:464] Registering TS with master...
I20260924 09:57:09.800804 11179 heartbeater.cc:511] Master 127.0.97.62:42477 requested a full tablet report, sending...
I20260924 09:57:09.802168 10723 ts_manager.cc:194] Registered new tserver with Master: c74150c79b9747bdb31c6c44093f58af (127.0.97.3:39847)
I20260924 09:57:09.802937 10723 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:54767
I20260924 09:57:09.813928 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 09:57:09.828787 10914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:57:09.818024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54410 (local address 127.0.97.62:42477)
0924 09:57:09.818219 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0924 09:57:09.818225 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 09:57:09.818558 (+ 333us) server_negotiation.cc:407] Connection header received
0924 09:57:09.818670 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:57:09.818673 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:57:09.818744 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:57:09.818819 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:57:09.820127 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:57:09.820842 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:57:09.821953 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:57:09.822338 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:57:09.822424 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:57:09.824411 (+ 1987us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:57:09.824415 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:57:09.824418 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:57:09.824449 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:57:09.826543 (+ 2094us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:57:09.827072 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:57:09.827075 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:57:09.827077 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:57:09.827184 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:57:09.827485 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:57:09.827487 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:57:09.827490 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:57:09.827758 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:57:09.827961 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0924 09:57:09.828333 (+ 372us) server_negotiation.cc:299] Negotiation successful
0924 09:57:09.828519 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 09:57:09.832463 10723 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54410:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 09:57:09.838982 10723 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 09:57:09.860502 11192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:57:09.850171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35974 (local address 127.0.97.3:39847)
0924 09:57:09.851889 (+ 1718us) server_negotiation.cc:206] Beginning negotiation
0924 09:57:09.851897 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 09:57:09.851927 (+ 30us) server_negotiation.cc:407] Connection header received
0924 09:57:09.852014 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:57:09.852022 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:57:09.852262 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:57:09.853196 (+ 934us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 09:57:09.853443 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:57:09.854444 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:57:09.857359 (+ 2915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:57:09.858617 (+ 1258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:57:09.858731 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:57:09.858958 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0924 09:57:09.859559 (+ 601us) server_negotiation.cc:299] Negotiation successful
0924 09:57:09.859715 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1565,"thread_start_us":2263,"threads_started":1}
I20260924 09:57:09.861224 11191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:57:09.849961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:39847 (local address 127.0.0.1:35974)
0924 09:57:09.850569 (+ 608us) negotiation.cc:107] Waiting for socket to connect
0924 09:57:09.850605 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 09:57:09.850752 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:57:09.852616 (+ 1864us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:57:09.852623 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:57:09.852660 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 09:57:09.853004 (+ 344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:57:09.853016 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:57:09.854573 (+ 1557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:57:09.854579 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:57:09.857206 (+ 2627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:57:09.857220 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:57:09.859091 (+ 1871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:57:09.859097 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:57:09.859323 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:57:09.859374 (+ 51us) client_negotiation.cc:769] Sending connection context
0924 09:57:09.860697 (+ 1323us) client_negotiation.cc:240] Negotiation successful
0924 09:57:09.860887 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":1014,"threads_started":1}
I20260924 09:57:09.862668 11193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:57:09.850816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38181 (local address 127.0.0.1:42532)
0924 09:57:09.854951 (+ 4135us) negotiation.cc:107] Waiting for socket to connect
0924 09:57:09.854972 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 09:57:09.855071 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:57:09.856179 (+ 1108us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:57:09.856184 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:57:09.856212 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 09:57:09.856639 (+ 427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:57:09.856650 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:57:09.858274 (+ 1624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:57:09.858279 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:57:09.860235 (+ 1956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:57:09.860257 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:57:09.861975 (+ 1718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:57:09.861981 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:57:09.862134 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:57:09.862188 (+ 54us) client_negotiation.cc:769] Sending connection context
0924 09:57:09.862261 (+ 73us) client_negotiation.cc:240] Negotiation successful
0924 09:57:09.862374 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4015,"thread_start_us":3872,"threads_started":1}
I20260924 09:57:09.862766 11194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:57:09.851388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42532 (local address 127.0.97.1:38181)
0924 09:57:09.855597 (+ 4209us) server_negotiation.cc:206] Beginning negotiation
0924 09:57:09.855605 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 09:57:09.855629 (+ 24us) server_negotiation.cc:407] Connection header received
0924 09:57:09.855708 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:57:09.855715 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:57:09.855923 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:57:09.856810 (+ 887us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 09:57:09.857007 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:57:09.858145 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:57:09.860388 (+ 2243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:57:09.861800 (+ 1412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:57:09.861906 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:57:09.862120 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0924 09:57:09.862315 (+ 195us) server_negotiation.cc:299] Negotiation successful
0924 09:57:09.862463 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4040,"thread_start_us":3899,"threads_started":1}
I20260924 09:57:09.863665 11196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:57:09.855871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:38047 (local address 127.0.0.1:56654)
0924 09:57:09.856438 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0924 09:57:09.856458 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 09:57:09.856676 (+ 218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:57:09.857429 (+ 753us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:57:09.857433 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:57:09.857455 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 09:57:09.857760 (+ 305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:57:09.857770 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:57:09.859472 (+ 1702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:57:09.859477 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:57:09.861118 (+ 1641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:57:09.861132 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:57:09.862970 (+ 1838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:57:09.862976 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:57:09.863130 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:57:09.863162 (+ 32us) client_negotiation.cc:769] Sending connection context
0924 09:57:09.863279 (+ 117us) client_negotiation.cc:240] Negotiation successful
0924 09:57:09.863397 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":292,"threads_started":1}
I20260924 09:57:09.863847 11195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:57:09.855356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56654 (local address 127.0.97.2:38047)
0924 09:57:09.855832 (+ 476us) server_negotiation.cc:206] Beginning negotiation
0924 09:57:09.855840 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 09:57:09.856557 (+ 717us) server_negotiation.cc:407] Connection header received
0924 09:57:09.856857 (+ 300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:57:09.856864 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:57:09.857069 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:57:09.857204 (+ 135us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 09:57:09.857942 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:57:09.859323 (+ 1381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:57:09.861287 (+ 1964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:57:09.862574 (+ 1287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:57:09.862687 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:57:09.862851 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0924 09:57:09.863319 (+ 468us) server_negotiation.cc:299] Negotiation successful
0924 09:57:09.863466 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":236,"threads_started":1}
I20260924 09:57:09.864929 11114 tablet_service.cc:1467] Processing CreateTablet for tablet 5217a6b3d4814f57a3bb90b0bda6a99d (DEFAULT_TABLE table=test-table [id=6bb9143522fa4076b4c5815afb867aa8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 09:57:09.866328 11114 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5217a6b3d4814f57a3bb90b0bda6a99d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 09:57:09.865931 10847 tablet_service.cc:1467] Processing CreateTablet for tablet 5217a6b3d4814f57a3bb90b0bda6a99d (DEFAULT_TABLE table=test-table [id=6bb9143522fa4076b4c5815afb867aa8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 09:57:09.867002 10847 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5217a6b3d4814f57a3bb90b0bda6a99d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 09:57:09.868378 10981 tablet_service.cc:1467] Processing CreateTablet for tablet 5217a6b3d4814f57a3bb90b0bda6a99d (DEFAULT_TABLE table=test-table [id=6bb9143522fa4076b4c5815afb867aa8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 09:57:09.869526 10981 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5217a6b3d4814f57a3bb90b0bda6a99d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 09:57:09.876355 10912 heartbeater.cc:503] Master 127.0.97.62:42477 was elected leader, sending a full tablet report...
I20260924 09:57:09.881692 11198 tablet_bootstrap.cc:492] T 5217a6b3d4814f57a3bb90b0bda6a99d P c74150c79b9747bdb31c6c44093f58af: Bootstrap starting.
I20260924 09:57:09.882833 11197 tablet_bootstrap.cc:492] T 5217a6b3d4814f57a3bb90b0bda6a99d P 22c41fefcd404cb387906261d16c8f42: Bootstrap starting.
I20260924 09:57:09.884621 11199 tablet_bootstrap.cc:492] T 5217a6b3d4814f57a3bb90b0bda6a99d P ac74072ff0ca4272937a2603959a2dab: Bootstrap starting.
I20260924 09:57:09.885514 11198 tablet_bootstrap.cc:654] T 5217a6b3d4814f57a3bb90b0bda6a99d P c74150c79b9747bdb31c6c44093f58af: Neither blocks nor log segments found. Creating new log.
I20260924 09:57:09.886451 11197 tablet_bootstrap.cc:654] T 5217a6b3d4814f57a3bb90b0bda6a99d P 22c41fefcd404cb387906261d16c8f42: Neither blocks nor log segments found. Creating new log.
I20260924 09:57:09.886695 11198 log.cc:824] T 5217a6b3d4814f57a3bb90b0bda6a99d P c74150c79b9747bdb31c6c44093f58af: Log is configured to *not* fsync() on all Append() calls
I20260924 09:57:09.887179 11199 tablet_bootstrap.cc:654] T 5217a6b3d4814f57a3bb90b0bda6a99d P ac74072ff0ca4272937a2603959a2dab: Neither blocks nor log segments found. Creating new log.
I20260924 09:57:09.887567 11197 log.cc:824] T 5217a6b3d4814f57a3bb90b0bda6a99d P 22c41fefcd404cb387906261d16c8f42: Log is configured to *not* fsync() on all Append() calls
I20260924 09:57:09.888270 11199 log.cc:824] T 5217a6b3d4814f57a3bb90b0bda6a99d P ac74072ff0ca4272937a2603959a2dab: Log is configured to *not* fsync() on all Append() calls
I20260924 09:57:09.889834 11198 tablet_bootstrap.cc:492] T 5217a6b3d4814f57a3bb90b0bda6a99d P c74150c79b9747bdb31c6c44093f58af: No bootstrap required, opened a new log
I20260924 09:57:09.890105 11198 ts_tablet_manager.cc:1397] T 5217a6b3d4814f57a3bb90b0bda6a99d P c74150c79b9747bdb31c6c44093f58af: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.002s
I20260924 09:57:09.893045 11199 tablet_bootstrap.cc:492] T 5217a6b3d4814f57a3bb90b0bda6a99d P ac74072ff0ca4272937a2603959a2dab: No bootstrap required, opened a new log
I20260924 09:57:09.893273 11199 ts_tablet_manager.cc:1397] T 5217a6b3d4814f57a3bb90b0bda6a99d P ac74072ff0ca4272937a2603959a2dab: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.004s
I20260924 09:57:09.895434 11197 tablet_bootstrap.cc:492] T 5217a6b3d4814f57a3bb90b0bda6a99d P 22c41fefcd404cb387906261d16c8f42: No bootstrap required, opened a new log
I20260924 09:57:09.895676 11197 ts_tablet_manager.cc:1397] T 5217a6b3d4814f57a3bb90b0bda6a99d P 22c41fefcd404cb387906261d16c8f42: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.007s
I20260924 09:57:09.899981 11198 raft_consensus.cc:348] T 5217a6b3d4814f57a3bb90b0bda6a99d P c74150c79b9747bdb31c6c44093f58af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c74150c79b9747bdb31c6c44093f58af" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39847 } } peers { permanent_uuid: "22c41fefcd404cb387906261d16c8f42" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38181 } } peers { permanent_uuid: "ac74072ff0ca4272937a2603959a2dab" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38047 } }
I20260924 09:57:09.900151 11199 raft_consensus.cc:348] T 5217a6b3d4814f57a3bb90b0bda6a99d P ac74072ff0ca4272937a2603959a2dab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c74150c79b9747bdb31c6c44093f58af" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39847 } } peers { permanent_uuid: "22c41fefcd404cb387906261d16c8f42" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38181 } } peers { permanent_uuid: "ac74072ff0ca4272937a2603959a2dab" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38047 } }
I20260924 09:57:09.900514 11199 raft_consensus.cc:374] T 5217a6b3d4814f57a3bb90b0bda6a99d P ac74072ff0ca4272937a2603959a2dab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 09:57:09.900641 11199 raft_consensus.cc:729] T 5217a6b3d4814f57a3bb90b0bda6a99d P ac74072ff0ca4272937a2603959a2dab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac74072ff0ca4272937a2603959a2dab, State: Initialized, Role: FOLLOWER
I20260924 09:57:09.900641 11198 raft_consensus.cc:374] T 5217a6b3d4814f57a3bb90b0bda6a99d P c74150c79b9747bdb31c6c44093f58af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 09:57:09.900789 11198 raft_consensus.cc:729] T 5217a6b3d4814f57a3bb90b0bda6a99d P c74150c79b9747bdb31c6c44093f58af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c74150c79b9747bdb31c6c44093f58af, State: Initialized, Role: FOLLOWER
I20260924 09:57:09.901218 11199 consensus_queue.cc:260] T 5217a6b3d4814f57a3bb90b0bda6a99d P ac74072ff0ca4272937a2603959a2dab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c74150c79b9747bdb31c6c44093f58af" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39847 } } peers { permanent_uuid: "22c41fefcd404cb387906261d16c8f42" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38181 } } peers { permanent_uuid: "ac74072ff0ca4272937a2603959a2dab" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38047 } }
I20260924 09:57:09.901450 11198 consensus_queue.cc:260] T 5217a6b3d4814f57a3bb90b0bda6a99d P c74150c79b9747bdb31c6c44093f58af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c74150c79b9747bdb31c6c44093f58af" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39847 } } peers { permanent_uuid: "22c41fefcd404cb387906261d16c8f42" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38181 } } peers { permanent_uuid: "ac74072ff0ca4272937a2603959a2dab" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38047 } }
I20260924 09:57:09.902171 11046 heartbeater.cc:503] Master 127.0.97.62:42477 was elected leader, sending a full tablet report...
I20260924 09:57:09.902604 11179 heartbeater.cc:503] Master 127.0.97.62:42477 was elected leader, sending a full tablet report...
I20260924 09:57:09.903332 11199 ts_tablet_manager.cc:1428] T 5217a6b3d4814f57a3bb90b0bda6a99d P ac74072ff0ca4272937a2603959a2dab: Time spent starting tablet: real 0.010s user 0.009s sys 0.000s
I20260924 09:57:09.905014 11198 ts_tablet_manager.cc:1428] T 5217a6b3d4814f57a3bb90b0bda6a99d P c74150c79b9747bdb31c6c44093f58af: Time spent starting tablet: real 0.015s user 0.009s sys 0.006s
I20260924 09:57:09.905367 11197 raft_consensus.cc:348] T 5217a6b3d4814f57a3bb90b0bda6a99d P 22c41fefcd404cb387906261d16c8f42 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c74150c79b9747bdb31c6c44093f58af" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39847 } } peers { permanent_uuid: "22c41fefcd404cb387906261d16c8f42" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38181 } } peers { permanent_uuid: "ac74072ff0ca4272937a2603959a2dab" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38047 } }
I20260924 09:57:09.905813 11197 raft_consensus.cc:374] T 5217a6b3d4814f57a3bb90b0bda6a99d P 22c41fefcd404cb387906261d16c8f42 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 09:57:09.905911 11197 raft_consensus.cc:729] T 5217a6b3d4814f57a3bb90b0bda6a99d P 22c41fefcd404cb387906261d16c8f42 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22c41fefcd404cb387906261d16c8f42, State: Initialized, Role: FOLLOWER
I20260924 09:57:09.906524 11197 consensus_queue.cc:260] T 5217a6b3d4814f57a3bb90b0bda6a99d P 22c41fefcd404cb387906261d16c8f42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c74150c79b9747bdb31c6c44093f58af" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39847 } } peers { permanent_uuid: "22c41fefcd404cb387906261d16c8f42" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38181 } } peers { permanent_uuid: "ac74072ff0ca4272937a2603959a2dab" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38047 } }
I20260924 09:57:09.908073 11197 ts_tablet_manager.cc:1428] T 5217a6b3d4814f57a3bb90b0bda6a99d P 22c41fefcd404cb387906261d16c8f42: Time spent starting tablet: real 0.012s user 0.007s sys 0.005s
I20260924 09:57:10.026799 11203 raft_consensus.cc:482] T 5217a6b3d4814f57a3bb90b0bda6a99d P ac74072ff0ca4272937a2603959a2dab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 09:57:10.027122 11203 raft_consensus.cc:504] T 5217a6b3d4814f57a3bb90b0bda6a99d P ac74072ff0ca4272937a2603959a2dab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c74150c79b9747bdb31c6c44093f58af" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39847 } } peers { permanent_uuid: "22c41fefcd404cb387906261d16c8f42" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38181 } } peers { permanent_uuid: "ac74072ff0ca4272937a2603959a2dab" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38047 } }
I20260924 09:57:10.028860 11203 leader_election.cc:288] T 5217a6b3d4814f57a3bb90b0bda6a99d P ac74072ff0ca4272937a2603959a2dab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c74150c79b9747bdb31c6c44093f58af (127.0.97.3:39847), 22c41fefcd404cb387906261d16c8f42 (127.0.97.1:38181)
W20260924 09:57:10.031483 11180 tablet.cc:2369] T 5217a6b3d4814f57a3bb90b0bda6a99d P c74150c79b9747bdb31c6c44093f58af: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 09:57:10.035976 11194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:57:10.029301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:35463 (local address 127.0.97.1:38181)
0924 09:57:10.029508 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0924 09:57:10.029514 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 09:57:10.030101 (+ 587us) server_negotiation.cc:407] Connection header received
0924 09:57:10.030242 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:57:10.030247 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:57:10.030318 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:57:10.030409 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 09:57:10.030966 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:57:10.031717 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:57:10.034319 (+ 2602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:57:10.035058 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:57:10.035191 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:57:10.035303 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0924 09:57:10.035629 (+ 326us) server_negotiation.cc:299] Negotiation successful
0924 09:57:10.035747 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260924 09:57:10.035996 11192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:57:10.029173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:35649 (local address 127.0.97.3:39847)
0924 09:57:10.029381 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0924 09:57:10.029387 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 09:57:10.029897 (+ 510us) server_negotiation.cc:407] Connection header received
0924 09:57:10.030095 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:57:10.030101 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:57:10.030170 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:57:10.030249 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 09:57:10.030862 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:57:10.031867 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:57:10.034288 (+ 2421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:57:10.035035 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:57:10.035129 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:57:10.035226 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0924 09:57:10.035655 (+ 429us) server_negotiation.cc:299] Negotiation successful
0924 09:57:10.035778 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260924 09:57:10.036160 11207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:57:10.029220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38181 (local address 127.0.97.2:35463)
0924 09:57:10.029992 (+ 772us) negotiation.cc:107] Waiting for socket to connect
0924 09:57:10.030005 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 09:57:10.030124 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:57:10.030418 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:57:10.030421 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:57:10.030454 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 09:57:10.030829 (+ 375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:57:10.030841 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:57:10.031840 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:57:10.031845 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:57:10.034136 (+ 2291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:57:10.034147 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:57:10.035308 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:57:10.035313 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:57:10.035486 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:57:10.035508 (+ 22us) client_negotiation.cc:769] Sending connection context
0924 09:57:10.035588 (+ 80us) client_negotiation.cc:240] Negotiation successful
0924 09:57:10.035704 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":670,"mutex_wait_us":83,"thread_start_us":492,"threads_started":1}
I20260924 09:57:10.036187 11206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:57:10.029063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:39847 (local address 127.0.97.2:35649)
0924 09:57:10.029767 (+ 704us) negotiation.cc:107] Waiting for socket to connect
0924 09:57:10.029793 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0924 09:57:10.029953 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:57:10.030275 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:57:10.030279 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:57:10.030322 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 09:57:10.030724 (+ 402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:57:10.030740 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:57:10.031987 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:57:10.031992 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:57:10.034133 (+ 2141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:57:10.034147 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:57:10.035275 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:57:10.035281 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:57:10.035472 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:57:10.035504 (+ 32us) client_negotiation.cc:769] Sending connection context
0924 09:57:10.035580 (+ 76us) client_negotiation.cc:240] Negotiation successful
0924 09:57:10.035709 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":523,"mutex_wait_us":33,"thread_start_us":323,"threads_started":1}
I20260924 09:57:10.037066 11134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5217a6b3d4814f57a3bb90b0bda6a99d" candidate_uuid: "ac74072ff0ca4272937a2603959a2dab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c74150c79b9747bdb31c6c44093f58af" is_pre_election: true
I20260924 09:57:10.037070 10867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5217a6b3d4814f57a3bb90b0bda6a99d" candidate_uuid: "ac74072ff0ca4272937a2603959a2dab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22c41fefcd404cb387906261d16c8f42" is_pre_election: true
I20260924 09:57:10.037513 11134 raft_consensus.cc:2454] T 5217a6b3d4814f57a3bb90b0bda6a99d P c74150c79b9747bdb31c6c44093f58af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac74072ff0ca4272937a2603959a2dab in term 0.
I20260924 09:57:10.037513 10867 raft_consensus.cc:2454] T 5217a6b3d4814f57a3bb90b0bda6a99d P 22c41fefcd404cb387906261d16c8f42 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac74072ff0ca4272937a2603959a2dab in term 0.
I20260924 09:57:10.038076 10935 leader_election.cc:302] T 5217a6b3d4814f57a3bb90b0bda6a99d P ac74072ff0ca4272937a2603959a2dab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac74072ff0ca4272937a2603959a2dab, c74150c79b9747bdb31c6c44093f58af; no voters:
I20260924 09:57:10.038429 11203 raft_consensus.cc:2781] T 5217a6b3d4814f57a3bb90b0bda6a99d P ac74072ff0ca4272937a2603959a2dab [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 09:57:10.038539 11203 raft_consensus.cc:482] T 5217a6b3d4814f57a3bb90b0bda6a99d P ac74072ff0ca4272937a2603959a2dab [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 09:57:10.038628 11203 raft_consensus.cc:3037] T 5217a6b3d4814f57a3bb90b0bda6a99d P ac74072ff0ca4272937a2603959a2dab [term 0 FOLLOWER]: Advancing to term 1
I20260924 09:57:10.040834 11203 raft_consensus.cc:504] T 5217a6b3d4814f57a3bb90b0bda6a99d P ac74072ff0ca4272937a2603959a2dab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c74150c79b9747bdb31c6c44093f58af" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39847 } } peers { permanent_uuid: "22c41fefcd404cb387906261d16c8f42" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38181 } } peers { permanent_uuid: "ac74072ff0ca4272937a2603959a2dab" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38047 } }
I20260924 09:57:10.041332 11203 leader_election.cc:288] T 5217a6b3d4814f57a3bb90b0bda6a99d P ac74072ff0ca4272937a2603959a2dab [CANDIDATE]: Term 1 election: Requested vote from peers c74150c79b9747bdb31c6c44093f58af (127.0.97.3:39847), 22c41fefcd404cb387906261d16c8f42 (127.0.97.1:38181)
I20260924 09:57:10.041682 11134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5217a6b3d4814f57a3bb90b0bda6a99d" candidate_uuid: "ac74072ff0ca4272937a2603959a2dab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c74150c79b9747bdb31c6c44093f58af"
I20260924 09:57:10.041788 10867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5217a6b3d4814f57a3bb90b0bda6a99d" candidate_uuid: "ac74072ff0ca4272937a2603959a2dab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22c41fefcd404cb387906261d16c8f42"
I20260924 09:57:10.041875 11134 raft_consensus.cc:3037] T 5217a6b3d4814f57a3bb90b0bda6a99d P c74150c79b9747bdb31c6c44093f58af [term 0 FOLLOWER]: Advancing to term 1
I20260924 09:57:10.041965 10867 raft_consensus.cc:3037] T 5217a6b3d4814f57a3bb90b0bda6a99d P 22c41fefcd404cb387906261d16c8f42 [term 0 FOLLOWER]: Advancing to term 1
I20260924 09:57:10.044899 11134 raft_consensus.cc:2454] T 5217a6b3d4814f57a3bb90b0bda6a99d P c74150c79b9747bdb31c6c44093f58af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac74072ff0ca4272937a2603959a2dab in term 1.
I20260924 09:57:10.044899 10867 raft_consensus.cc:2454] T 5217a6b3d4814f57a3bb90b0bda6a99d P 22c41fefcd404cb387906261d16c8f42 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac74072ff0ca4272937a2603959a2dab in term 1.
I20260924 09:57:10.045272 10934 leader_election.cc:302] T 5217a6b3d4814f57a3bb90b0bda6a99d P ac74072ff0ca4272937a2603959a2dab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22c41fefcd404cb387906261d16c8f42, ac74072ff0ca4272937a2603959a2dab; no voters:
I20260924 09:57:10.045557 11203 raft_consensus.cc:2781] T 5217a6b3d4814f57a3bb90b0bda6a99d P ac74072ff0ca4272937a2603959a2dab [term 1 FOLLOWER]: Leader election won for term 1
I20260924 09:57:10.045945 11203 raft_consensus.cc:686] T 5217a6b3d4814f57a3bb90b0bda6a99d P ac74072ff0ca4272937a2603959a2dab [term 1 LEADER]: Becoming Leader. State: Replica: ac74072ff0ca4272937a2603959a2dab, State: Running, Role: LEADER
I20260924 09:57:10.046348 11203 consensus_queue.cc:237] T 5217a6b3d4814f57a3bb90b0bda6a99d P ac74072ff0ca4272937a2603959a2dab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c74150c79b9747bdb31c6c44093f58af" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39847 } } peers { permanent_uuid: "22c41fefcd404cb387906261d16c8f42" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38181 } } peers { permanent_uuid: "ac74072ff0ca4272937a2603959a2dab" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38047 } }
I20260924 09:57:10.050268 10723 catalog_manager.cc:5495] T 5217a6b3d4814f57a3bb90b0bda6a99d P ac74072ff0ca4272937a2603959a2dab reported cstate change: term changed from 0 to 1, leader changed from <none> to ac74072ff0ca4272937a2603959a2dab (127.0.97.2). New cstate: current_term: 1 leader_uuid: "ac74072ff0ca4272937a2603959a2dab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c74150c79b9747bdb31c6c44093f58af" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39847 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22c41fefcd404cb387906261d16c8f42" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ac74072ff0ca4272937a2603959a2dab" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38047 } health_report { overall_health: HEALTHY } } }
W20260924 09:57:10.065980 11047 tablet.cc:2369] T 5217a6b3d4814f57a3bb90b0bda6a99d P ac74072ff0ca4272937a2603959a2dab: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 09:57:10.095913 11195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:57:10.091370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56656 (local address 127.0.97.2:38047)
0924 09:57:10.091530 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0924 09:57:10.091534 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 09:57:10.091587 (+ 53us) server_negotiation.cc:407] Connection header received
0924 09:57:10.091761 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:57:10.091765 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:57:10.091849 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:57:10.091959 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 09:57:10.092269 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:57:10.092971 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:57:10.094204 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:57:10.094561 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:57:10.094648 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:57:10.094999 (+ 351us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 09:57:10.095495 (+ 496us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 09:57:10.095591 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0924 09:57:10.095655 (+ 64us) server_negotiation.cc:299] Negotiation successful
0924 09:57:10.095711 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
W20260924 09:57:10.099713 10913 tablet.cc:2369] T 5217a6b3d4814f57a3bb90b0bda6a99d P 22c41fefcd404cb387906261d16c8f42: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 09:57:10.103214 11134 raft_consensus.cc:1261] T 5217a6b3d4814f57a3bb90b0bda6a99d P c74150c79b9747bdb31c6c44093f58af [term 1 FOLLOWER]: Refusing update from remote peer ac74072ff0ca4272937a2603959a2dab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 09:57:10.103227 10867 raft_consensus.cc:1261] T 5217a6b3d4814f57a3bb90b0bda6a99d P 22c41fefcd404cb387906261d16c8f42 [term 1 FOLLOWER]: Refusing update from remote peer ac74072ff0ca4272937a2603959a2dab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 09:57:10.104135 11208 consensus_queue.cc:1035] T 5217a6b3d4814f57a3bb90b0bda6a99d P ac74072ff0ca4272937a2603959a2dab [LEADER]: Connected to new peer: Peer: permanent_uuid: "c74150c79b9747bdb31c6c44093f58af" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 09:57:10.104439 11203 consensus_queue.cc:1035] T 5217a6b3d4814f57a3bb90b0bda6a99d P ac74072ff0ca4272937a2603959a2dab [LEADER]: Connected to new peer: Peer: permanent_uuid: "22c41fefcd404cb387906261d16c8f42" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 09:57:10.116664 11211 mvcc.cc:205] Tried to move back new op lower bound from 7332838728093536256 to 7332838727873060864. Current Snapshot: MvccSnapshot[applied={T|T < 7332838728093536256}]
I20260924 09:57:10.117748 11213 mvcc.cc:205] Tried to move back new op lower bound from 7332838728093536256 to 7332838727873060864. Current Snapshot: MvccSnapshot[applied={T|T < 7332838728093536256 or (T in {7332838728093536256})}]
I20260924 09:57:10.127830 10722 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54410:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 09:57:10.128082 10722 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54410:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 09:57:10.131258 10722 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 340849d406bd452b952004a0ae6c375b: Sending DeleteTablet for 3 replicas of tablet 5217a6b3d4814f57a3bb90b0bda6a99d
I20260924 09:57:10.132277 11114 tablet_service.cc:1514] Processing DeleteTablet for tablet 5217a6b3d4814f57a3bb90b0bda6a99d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 09:57:10 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:35974
I20260924 09:57:10.132365 10981 tablet_service.cc:1514] Processing DeleteTablet for tablet 5217a6b3d4814f57a3bb90b0bda6a99d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 09:57:10 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:56654
I20260924 09:57:10.133106 10847 tablet_service.cc:1514] Processing DeleteTablet for tablet 5217a6b3d4814f57a3bb90b0bda6a99d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 09:57:10 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:42532
I20260924 09:57:10.133353 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu pid 10784
I20260924 09:57:10.135761 11223 tablet_replica.cc:330] T 5217a6b3d4814f57a3bb90b0bda6a99d P 22c41fefcd404cb387906261d16c8f42: stopping tablet replica
I20260924 09:57:10.136201 11223 raft_consensus.cc:2229] T 5217a6b3d4814f57a3bb90b0bda6a99d P 22c41fefcd404cb387906261d16c8f42 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 09:57:10.136721 11223 raft_consensus.cc:2258] T 5217a6b3d4814f57a3bb90b0bda6a99d P 22c41fefcd404cb387906261d16c8f42 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 09:57:10.138579 11223 ts_tablet_manager.cc:1905] T 5217a6b3d4814f57a3bb90b0bda6a99d P 22c41fefcd404cb387906261d16c8f42: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 09:57:10.144667 11221 tablet_replica.cc:330] T 5217a6b3d4814f57a3bb90b0bda6a99d P c74150c79b9747bdb31c6c44093f58af: stopping tablet replica
I20260924 09:57:10.145207 11222 tablet_replica.cc:330] T 5217a6b3d4814f57a3bb90b0bda6a99d P ac74072ff0ca4272937a2603959a2dab: stopping tablet replica
I20260924 09:57:10.145308 11223 ts_tablet_manager.cc:1918] T 5217a6b3d4814f57a3bb90b0bda6a99d P 22c41fefcd404cb387906261d16c8f42: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 09:57:10.145536 11223 log.cc:1196] T 5217a6b3d4814f57a3bb90b0bda6a99d P 22c41fefcd404cb387906261d16c8f42: Deleting WAL directory at /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790243735634835-388-0/minicluster-data/ts-0/wal/wals/5217a6b3d4814f57a3bb90b0bda6a99d
I20260924 09:57:10.145727 11221 raft_consensus.cc:2229] T 5217a6b3d4814f57a3bb90b0bda6a99d P c74150c79b9747bdb31c6c44093f58af [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 09:57:10.146091 11223 ts_tablet_manager.cc:1939] T 5217a6b3d4814f57a3bb90b0bda6a99d P 22c41fefcd404cb387906261d16c8f42: Deleting consensus metadata
I20260924 09:57:10.146202 11222 raft_consensus.cc:2229] T 5217a6b3d4814f57a3bb90b0bda6a99d P ac74072ff0ca4272937a2603959a2dab [term 1 LEADER]: Raft consensus shutting down.
I20260924 09:57:10.146927 11222 raft_consensus.cc:2258] T 5217a6b3d4814f57a3bb90b0bda6a99d P ac74072ff0ca4272937a2603959a2dab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 09:57:10.147452 11221 raft_consensus.cc:2258] T 5217a6b3d4814f57a3bb90b0bda6a99d P c74150c79b9747bdb31c6c44093f58af [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 09:57:10.147920 10707 catalog_manager.cc:4841] TS 22c41fefcd404cb387906261d16c8f42 (127.0.97.1:38181): tablet 5217a6b3d4814f57a3bb90b0bda6a99d (table test-table [id=6bb9143522fa4076b4c5815afb867aa8]) successfully deleted
I20260924 09:57:10.149355 11221 ts_tablet_manager.cc:1905] T 5217a6b3d4814f57a3bb90b0bda6a99d P c74150c79b9747bdb31c6c44093f58af: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 09:57:10.149521 11222 ts_tablet_manager.cc:1905] T 5217a6b3d4814f57a3bb90b0bda6a99d P ac74072ff0ca4272937a2603959a2dab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 09:57:10.155537 11221 ts_tablet_manager.cc:1918] T 5217a6b3d4814f57a3bb90b0bda6a99d P c74150c79b9747bdb31c6c44093f58af: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 09:57:10.155730 11221 log.cc:1196] T 5217a6b3d4814f57a3bb90b0bda6a99d P c74150c79b9747bdb31c6c44093f58af: Deleting WAL directory at /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790243735634835-388-0/minicluster-data/ts-2/wal/wals/5217a6b3d4814f57a3bb90b0bda6a99d
I20260924 09:57:10.156294 11221 ts_tablet_manager.cc:1939] T 5217a6b3d4814f57a3bb90b0bda6a99d P c74150c79b9747bdb31c6c44093f58af: Deleting consensus metadata
I20260924 09:57:10.156440 11222 ts_tablet_manager.cc:1918] T 5217a6b3d4814f57a3bb90b0bda6a99d P ac74072ff0ca4272937a2603959a2dab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 09:57:10.156636 11222 log.cc:1196] T 5217a6b3d4814f57a3bb90b0bda6a99d P ac74072ff0ca4272937a2603959a2dab: Deleting WAL directory at /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790243735634835-388-0/minicluster-data/ts-1/wal/wals/5217a6b3d4814f57a3bb90b0bda6a99d
I20260924 09:57:10.157186 11222 ts_tablet_manager.cc:1939] T 5217a6b3d4814f57a3bb90b0bda6a99d P ac74072ff0ca4272937a2603959a2dab: Deleting consensus metadata
I20260924 09:57:10.158012 10708 catalog_manager.cc:4841] TS c74150c79b9747bdb31c6c44093f58af (127.0.97.3:39847): tablet 5217a6b3d4814f57a3bb90b0bda6a99d (table test-table [id=6bb9143522fa4076b4c5815afb867aa8]) successfully deleted
I20260924 09:57:10.158774 10707 catalog_manager.cc:4841] TS ac74072ff0ca4272937a2603959a2dab (127.0.97.2:38047): tablet 5217a6b3d4814f57a3bb90b0bda6a99d (table test-table [id=6bb9143522fa4076b4c5815afb867aa8]) successfully deleted
Sep 24 09:57:10 dist-test-slave-w8d2 krb5kdc[10678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243827, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260924 09:57:10.167511 11194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:57:10.134411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42544 (local address 127.0.97.1:38181)
0924 09:57:10.134568 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0924 09:57:10.134573 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 09:57:10.152967 (+ 18394us) server_negotiation.cc:407] Connection header received
0924 09:57:10.153394 (+ 427us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:57:10.153399 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:57:10.153479 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:57:10.153544 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:57:10.155124 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:57:10.156187 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:57:10.157910 (+ 1723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:57:10.158437 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:57:10.158507 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:57:10.162455 (+ 3948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:57:10.162466 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:57:10.162479 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:57:10.162508 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:57:10.165069 (+ 2561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:57:10.165788 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:57:10.165794 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:57:10.165797 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:57:10.165903 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:57:10.166255 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:57:10.166258 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:57:10.166259 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:57:10.166559 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:57:10.166753 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0924 09:57:10.167061 (+ 308us) server_negotiation.cc:299] Negotiation successful
0924 09:57:10.167214 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 09:57:10.168115 10907 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 09:57:10.273838 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu with pid 10784
I20260924 09:57:10.287984 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu pid 10918
Sep 24 09:57:10 dist-test-slave-w8d2 krb5kdc[10678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243827, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260924 09:57:10.306327 11195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:57:10.288697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56658 (local address 127.0.97.2:38047)
0924 09:57:10.288882 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0924 09:57:10.288888 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 09:57:10.289001 (+ 113us) server_negotiation.cc:407] Connection header received
0924 09:57:10.289183 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:57:10.289187 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:57:10.289280 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:57:10.289381 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:57:10.290936 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:57:10.292129 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:57:10.293955 (+ 1826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:57:10.294605 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:57:10.294717 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:57:10.299738 (+ 5021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:57:10.299756 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:57:10.299775 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:57:10.299820 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:57:10.303558 (+ 3738us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:57:10.304295 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:57:10.304305 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:57:10.304308 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:57:10.304473 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:57:10.304810 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:57:10.304815 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:57:10.304819 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:57:10.305244 (+ 425us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:57:10.305547 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0924 09:57:10.305754 (+ 207us) server_negotiation.cc:299] Negotiation successful
0924 09:57:10.305955 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260924 09:57:10.307015 11041 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 09:57:10.420802 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu with pid 10918
I20260924 09:57:10.438304 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu pid 11051
Sep 24 09:57:10 dist-test-slave-w8d2 krb5kdc[10678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243827, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260924 09:57:10.451457 11192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:57:10.438993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35990 (local address 127.0.97.3:39847)
0924 09:57:10.439164 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0924 09:57:10.439168 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 09:57:10.439186 (+ 18us) server_negotiation.cc:407] Connection header received
0924 09:57:10.439388 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:57:10.439391 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:57:10.439455 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:57:10.439546 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:57:10.440655 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:57:10.441355 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:57:10.442492 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:57:10.442861 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:57:10.442947 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:57:10.446704 (+ 3757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:57:10.446718 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:57:10.446734 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:57:10.446764 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:57:10.449220 (+ 2456us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:57:10.449742 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:57:10.449748 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:57:10.449751 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:57:10.449867 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:57:10.450149 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:57:10.450152 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:57:10.450154 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:57:10.450490 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:57:10.450684 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0924 09:57:10.450928 (+ 244us) server_negotiation.cc:299] Negotiation successful
0924 09:57:10.451075 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 09:57:10.452040 11174 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 09:57:10.538219 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu with pid 11051
I20260924 09:57:10.550904 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu pid 10691
I20260924 09:57:10.551967 10753 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 09:57:10.654753 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu with pid 10691
2026-09-24T09:57:10Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (3991 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790243735634835-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 09:57:10 dist-test-slave-w8d2 krb5kdc[11231](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 09:57:10 dist-test-slave-w8d2 krb5kdc[11231](info): set up 2 sockets
Sep 24 09:57:10 dist-test-slave-w8d2 krb5kdc[11231](info): commencing operation
krb5kdc: starting...
W20260924 09:57:11.651880 388 mini_kdc.cc:121] Time spent starting KDC: real 0.960s user 0.008s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790243735634835-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790243735634835-388-0/krb5kdc/test-user.keytab.
Sep 24 09:57:11 dist-test-slave-w8d2 krb5kdc[11231](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243831, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T09:57:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T09:57:11Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790243735634835-388-0/krb5kdc/rangeradmin_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790243735634835-388-0/krb5kdc/rangeradmin_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790243735634835-388-0/krb5kdc/rangerlookup_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790243735634835-388-0/krb5kdc/rangerlookup_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790243735634835-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790243735634835-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
I20260924 09:57:11.925812 388 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790243735634835-388-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskPt_ZmK/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790243735634835-388-0/minicluster-data/postgres -l logfile start
2026-09-24 09:57:14.256 UTC [11261] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-24 09:57:14.256 UTC [11261] LOG: listening on IPv4 address "127.0.97.20", port 47869
2026-09-24 09:57:14.259 UTC [11261] LOG: listening on Unix socket "/tmp/.s.PGSQL.47869"
2026-09-24 09:57:14.279 UTC [11264] LOG: database system was shut down at 2026-09-24 09:57:12 UTC
2026-09-24 09:57:14.282 UTC [11261] LOG: database system is ready to accept connections
I20260924 09:57:15.163883 388 mini_postgres.cc:96] Postgres bound to 47869
2026-09-24 09:57:15.179 UTC [11272] FATAL: database "slave" does not exist
127.0.97.20:47869 - accepting connections
I20260924 09:57:15.180399 388 mini_ranger.cc:161] Starting Ranger...
I20260924 09:57:15.204423 388 mini_ranger.cc:84] Created miniranger Postgres user
I20260924 09:57:15.658870 388 mini_ranger.cc:87] Created ranger Postgres database
I20260924 09:57:15.659039 388 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790243735634835-388-0/minicluster-data/ranger-admin
2026-09-24 09:57:15,940 [I] DB FLAVOR :POSTGRES
2026-09-24 09:57:15,941 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 09:57:15,941 [I] Checking connection..
2026-09-24 09:57:15,941 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPt_ZmK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskPt_ZmK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:47869/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-24 09:57:16,194 [I] Checking connection passed.
2026-09-24 09:57:16,194 [I] --------- Verifying version history table ---------
2026-09-24 09:57:16,194 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPt_ZmK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskPt_ZmK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:47869/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 09:57:16,427 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 09:57:16,428 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPt_ZmK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskPt_ZmK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:47869/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 09:57:16,670 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 09:57:16,670 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-24 09:57:16,671 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPt_ZmK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskPt_ZmK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:47869/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskPt_ZmK/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-24 09:57:16.918 UTC [11336] WARNING: there is no transaction in progress
2026-09-24 09:57:16,923 [I] create_dbversion_catalog.sql file imported successfully
2026-09-24 09:57:16,923 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPt_ZmK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskPt_ZmK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:47869/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 09:57:17,159 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-24 09:57:17,159 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 09:57:17,159 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPt_ZmK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskPt_ZmK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:47869/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-24 09:57:17,384 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPt_ZmK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskPt_ZmK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:47869/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-24 09:57:17,618 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPt_ZmK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskPt_ZmK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:47869/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-w8d2.c.gcp-upstream.internal','N') ;"
2026-09-24 09:57:17,860 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPt_ZmK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskPt_ZmK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:47869/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 09:57:18,110 [I] Table x_portal_user does not exist in database ranger
2026-09-24 09:57:18,110 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPt_ZmK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskPt_ZmK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:47869/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 09:57:18,364 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-24 09:57:18,365 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPt_ZmK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskPt_ZmK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:47869/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 09:57:18,604 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPt_ZmK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskPt_ZmK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:47869/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 09:57:18,831 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-24 09:57:18,831 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPt_ZmK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskPt_ZmK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:47869/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskPt_ZmK/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-24 09:57:19.747 UTC [11444] WARNING: there is no transaction in progress
2026-09-24 09:57:19.762 UTC [11444] WARNING: there is no transaction in progress
2026-09-24 09:57:19.777 UTC [11444] WARNING: there is no transaction in progress
2026-09-24 09:57:19.791 UTC [11444] WARNING: there is no transaction in progress
2026-09-24 09:57:19.806 UTC [11444] WARNING: there is no transaction in progress
2026-09-24 09:57:19.825 UTC [11444] WARNING: there is no transaction in progress
2026-09-24 09:57:19.878 UTC [11444] WARNING: there is no transaction in progress
2026-09-24 09:57:19.884 UTC [11444] WARNING: there is no transaction in progress
2026-09-24 09:57:19.896 UTC [11444] WARNING: there is no transaction in progress
2026-09-24 09:57:19.906 UTC [11444] WARNING: there is no transaction in progress
2026-09-24 09:57:19.915 UTC [11444] WARNING: there is no transaction in progress
2026-09-24 09:57:19.928 UTC [11444] WARNING: there is no transaction in progress
2026-09-24 09:57:19.936 UTC [11444] WARNING: there is no transaction in progress
2026-09-24 09:57:19.942 UTC [11444] WARNING: there is no transaction in progress
2026-09-24 09:57:19.949 UTC [11444] WARNING: there is no transaction in progress
2026-09-24 09:57:19.955 UTC [11444] WARNING: there is no transaction in progress
2026-09-24 09:57:20,516 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-24 09:57:20,516 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPt_ZmK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskPt_ZmK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:47869/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-24 09:57:20,761 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-24 09:57:20,762 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPt_ZmK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskPt_ZmK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:47869/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 09:57:21,010 [I] Table x_portal_user already exists in database 'ranger'
2026-09-24 09:57:21,010 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPt_ZmK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskPt_ZmK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:47869/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 09:57:21,252 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-24 09:57:21,252 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPt_ZmK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskPt_ZmK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:47869/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 09:57:21,502 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPt_ZmK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskPt_ZmK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:47869/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 09:57:21,733 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPt_ZmK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskPt_ZmK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:47869/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-w8d2.c.gcp-upstream.internal';"
2026-09-24 09:57:21,973 [I] CORE_DB_SCHEMA import status has been updated
2026-09-24 09:57:21,973 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPt_ZmK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskPt_ZmK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:47869/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-24 09:57:22,210 [I] DB_PATCHES have already been applied
I20260924 09:57:22.215736 388 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790243735634835-388-0/minicluster-data/ranger-admin:/tmp/dist-test-taskPt_ZmK/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskPt_ZmK/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskPt_ZmK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskPt_ZmK/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260924 09:57:22.215835 388 mini_ranger.cc:193] Using host: 127.0.97.20
I20260924 09:57:22.228669 388 mini_ranger.cc:227] Ranger admin URL: http://127.0.97.20:43439
Sep 24, 2026 9:57:22 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 9:57:22 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 9:57:22 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790243735634835-388-0/minicluster-data/ranger-admin/ews/webapp
Sep 24, 2026 9:57:22 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790243735634835-388-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 24, 2026 9:57:22 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790243735634835-388-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 24, 2026 9:57:23 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790243735634835-388-0/minicluster-data/ranger-admin/ews/webapp].
Sep 24, 2026 9:57:23 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260924 09:57:23.229736 388 mini_ranger.cc:160] Time spent starting Ranger: real 8.049s user 0.007s sys 0.028s
Sep 24, 2026 9:57:41 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 24, 2026 9:57:41 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 24, 2026 9:57:41 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 24, 2026 9:57:41 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 24, 2026 9:57:41 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 24, 2026 9:57:41 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 24, 2026 9:57:41 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 24, 2026 9:57:41 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 24, 2026 9:57:41 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 24, 2026 9:57:41 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 24, 2026 9:57:41 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 24, 2026 9:57:41 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 24, 2026 9:57:41 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 24, 2026 9:57:41 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 24, 2026 9:57:41 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 24, 2026 9:57:41 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 24, 2026 9:57:41 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 24, 2026 9:57:42 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260924 09:57:43.319180 388 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790243735634835-388-0/krb5kdc/rangerkms_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790243735634835-388-0/krb5kdc/rangerkms_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.97.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.97.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790243735634835-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790243735634835-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 24 09:57:43 dist-test-slave-w8d2 krb5kdc[11231](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243863, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260924 09:57:43.465292 388 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260924 09:57:43.482622 388 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260924 09:57:44.203734 388 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260924 09:57:44.203855 388 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790243735634835-388-0/minicluster-data/ranger-kms
I20260924 09:57:44.203995 388 mini_ranger_kms.cc:227] Using postgres at 127.0.97.20:47869
2026-09-24 09:57:44,576 [I] DB FLAVOR :POSTGRES
2026-09-24 09:57:44,576 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 09:57:44,576 [I] Checking connection
2026-09-24 09:57:44,576 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPt_ZmK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790243735634835-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:47869/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-24 09:57:44,809 [I] connection success
2026-09-24 09:57:44,809 [I] --------- Verifying Ranger DB tables ---------
2026-09-24 09:57:44,809 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-24 09:57:44,809 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPt_ZmK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790243735634835-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:47869/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-24 09:57:45,048 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-24 09:57:45,049 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 09:57:45,049 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-24 09:57:45,049 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPt_ZmK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790243735634835-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:47869/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790243735634835-388-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-24 09:57:45,285 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260924 09:57:45.706578 388 mini_ranger_kms.cc:310] Created kms service
I20260924 09:57:45.873137 388 mini_ranger_kms.cc:326] Created kudu user
I20260924 09:57:45.913789 388 mini_ranger_kms.cc:343] Created rangerkms user
Sep 24 09:57:45 dist-test-slave-w8d2 krb5kdc[11231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243863, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260924 09:57:46.306253 388 mini_ranger_kms.cc:384] Added ranger policy
I20260924 09:57:46.306540 388 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790243735634835-388-0/minicluster-data/ranger-kms:/tmp/dist-test-taskPt_ZmK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskPt_ZmK/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskPt_ZmK/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260924 09:57:46.306594 388 mini_ranger_kms.cc:240] Using host: 127.0.97.20
I20260924 09:57:46.330004 388 mini_ranger_kms.cc:276] Ranger KMS PID: 11624
I20260924 09:57:46.330163 388 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.97.20:43821
Sep 24, 2026 9:57:47 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 9:57:47 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 9:57:47 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790243735634835-388-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 24, 2026 9:57:47 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790243735634835-388-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 24, 2026 9:57:47 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790243735634835-388-0/minicluster-data/ranger-kms/ews/webapp].
Sep 24, 2026 9:57:47 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260924 09:57:48.363056 388 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 4.898s user 0.111s sys 0.279s
I20260924 09:57:48.363329 388 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260924 09:57:48.363523 388 mini_ranger_kms.cc:401] 127.0.97.20:43821/kms/v1/keys
Sep 24 09:57:48 dist-test-slave-w8d2 krb5kdc[11231](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790243868, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 24 09:57:48 dist-test-slave-w8d2 krb5kdc[11231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243868, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Sep 24 09:57:49 dist-test-slave-w8d2 krb5kdc[11231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243868, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Sep 24 09:57:51 dist-test-slave-w8d2 krb5kdc[11231](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243871, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790243735634835-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790243735634835-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260924 09:57:51.542115 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
/tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790243735634835-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790243735634835-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790243735634835-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790243735634835-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40811
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790243735634835-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:40811
--ranger_config_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790243735634835-388-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:43821/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790243735634835-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790243735634835-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790243735634835-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 09:57:51.857226 11675 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 09:57:51.857504 11675 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 09:57:51.857589 11675 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 09:57:51.865084 11675 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 09:57:51.865218 11675 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 09:57:51.865293 11675 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 09:57:51.865366 11675 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 09:57:51.865428 11675 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 09:57:51.875396 11675 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790243735634835-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790243735634835-388-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.97.20:43821/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:40811
--ranger_config_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790243735634835-388-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790243735634835-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40811
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790243735634835-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790243735634835-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 09:57:51.876611 11675 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 09:57:51.877813 11675 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 09:57:51.889755 11681 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:57:51.889807 11683 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:57:51.889734 11680 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 09:57:51.890367 11675 server_base.cc:1034] running on GCE node
I20260924 09:57:51.891114 11675 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 09:57:51.892556 11675 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 09:57:51.893741 11675 hybrid_clock.cc:648] HybridClock initialized: now 1790243871893698 us; error 34 us; skew 500 ppm
I20260924 09:57:51.894068 11675 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 09:57:51 dist-test-slave-w8d2 krb5kdc[11231](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243871, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 09:57:51.899915 11675 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260924 09:57:51.901360 11675 webserver.cc:460] Webserver started at http://127.0.97.62:35199/ using document root <none> and password file <none>
I20260924 09:57:51.901966 11675 fs_manager.cc:359] Metadata directory not provided
I20260924 09:57:51.902064 11675 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 09:57:51.902298 11675 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 09:57:51 dist-test-slave-w8d2 krb5kdc[11231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243871, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260924 09:57:51.949213 11675 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790243735634835-388-0/minicluster-data/master-0/data/instance:
uuid: "42a9029a7aec461eb55f998a506b41f7"
format_stamp: "Formatted at 2026-09-24 09:57:51 on dist-test-slave-w8d2"
server_key: "3af004d5f852c25eeb6e14178a5c2968"
server_key_iv: "47bc9285890e9eb697c91d501b778d96"
server_key_version: "kuduclusterkey@0"
I20260924 09:57:51.949991 11675 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790243735634835-388-0/minicluster-data/master-0/wal/instance:
uuid: "42a9029a7aec461eb55f998a506b41f7"
format_stamp: "Formatted at 2026-09-24 09:57:51 on dist-test-slave-w8d2"
server_key: "3af004d5f852c25eeb6e14178a5c2968"
server_key_iv: "47bc9285890e9eb697c91d501b778d96"
server_key_version: "kuduclusterkey@0"
I20260924 09:57:51.955166 11675 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260924 09:57:51.970865 11690 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 09:57:51.972352 11675 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260924 09:57:51.972522 11675 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790243735634835-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790243735634835-388-0/minicluster-data/master-0/wal
uuid: "42a9029a7aec461eb55f998a506b41f7"
format_stamp: "Formatted at 2026-09-24 09:57:51 on dist-test-slave-w8d2"
server_key: "3af004d5f852c25eeb6e14178a5c2968"
server_key_iv: "47bc9285890e9eb697c91d501b778d96"
server_key_version: "kuduclusterkey@0"
I20260924 09:57:51.972723 11675 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790243735634835-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790243735634835-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790243735634835-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 09:57:52.007056 11675 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 09:57:52.010875 11675 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 09:57:52.011142 11675 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 09:57:52.030088 11675 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:40811
I20260924 09:57:52.030097 11741 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:40811 every 8 connection(s)
I20260924 09:57:52.031419 11675 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790243735634835-388-0/minicluster-data/master-0/data/info.pb
I20260924 09:57:52.034971 11742 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 09:57:52.035630 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu as pid 11675
I20260924 09:57:52.035823 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790243735634835-388-0/minicluster-data/master-0/wal/instance
I20260924 09:57:52.036620 388 external_mini_cluster.cc:1428] Setting key 10da2effd278e874c1443e3da0760342
I20260924 09:57:52.050319 11742 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 42a9029a7aec461eb55f998a506b41f7: Bootstrap starting.
I20260924 09:57:52.053575 11742 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 42a9029a7aec461eb55f998a506b41f7: Neither blocks nor log segments found. Creating new log.
I20260924 09:57:52.054467 11742 log.cc:824] T 00000000000000000000000000000000 P 42a9029a7aec461eb55f998a506b41f7: Log is configured to *not* fsync() on all Append() calls
Sep 24 09:57:52 dist-test-slave-w8d2 krb5kdc[11231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243871, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 09:57:52.057562 11742 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 42a9029a7aec461eb55f998a506b41f7: No bootstrap required, opened a new log
I20260924 09:57:52.064420 11745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:57:52.038654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60414 (local address 127.0.97.62:40811)
0924 09:57:52.043906 (+ 5252us) server_negotiation.cc:206] Beginning negotiation
0924 09:57:52.043927 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0924 09:57:52.044473 (+ 546us) server_negotiation.cc:407] Connection header received
0924 09:57:52.046183 (+ 1710us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:57:52.046211 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:57:52.046731 (+ 520us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:57:52.047189 (+ 458us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:57:52.048241 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:57:52.050071 (+ 1830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:57:52.051322 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:57:52.052313 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:57:52.052484 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:57:52.056200 (+ 3716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:57:52.056210 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:57:52.056228 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:57:52.056272 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:57:52.060266 (+ 3994us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:57:52.060873 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:57:52.060884 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:57:52.060893 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:57:52.061082 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:57:52.061376 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:57:52.061381 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:57:52.061384 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:57:52.062010 (+ 626us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:57:52.062292 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0924 09:57:52.062691 (+ 399us) server_negotiation.cc:299] Negotiation successful
0924 09:57:52.063100 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4938,"thread_start_us":4698,"threads_started":1}
I20260924 09:57:52.066927 11742 raft_consensus.cc:348] T 00000000000000000000000000000000 P 42a9029a7aec461eb55f998a506b41f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42a9029a7aec461eb55f998a506b41f7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40811 } }
I20260924 09:57:52.067261 11742 raft_consensus.cc:374] T 00000000000000000000000000000000 P 42a9029a7aec461eb55f998a506b41f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 09:57:52.067345 11742 raft_consensus.cc:729] T 00000000000000000000000000000000 P 42a9029a7aec461eb55f998a506b41f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42a9029a7aec461eb55f998a506b41f7, State: Initialized, Role: FOLLOWER
I20260924 09:57:52.067804 11742 consensus_queue.cc:260] T 00000000000000000000000000000000 P 42a9029a7aec461eb55f998a506b41f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42a9029a7aec461eb55f998a506b41f7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40811 } }
I20260924 09:57:52.067991 11742 raft_consensus.cc:388] T 00000000000000000000000000000000 P 42a9029a7aec461eb55f998a506b41f7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 09:57:52.068079 11742 raft_consensus.cc:482] T 00000000000000000000000000000000 P 42a9029a7aec461eb55f998a506b41f7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 09:57:52.068187 11742 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 42a9029a7aec461eb55f998a506b41f7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 09:57:52.070091 11742 raft_consensus.cc:504] T 00000000000000000000000000000000 P 42a9029a7aec461eb55f998a506b41f7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42a9029a7aec461eb55f998a506b41f7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40811 } }
I20260924 09:57:52.070497 11742 leader_election.cc:302] T 00000000000000000000000000000000 P 42a9029a7aec461eb55f998a506b41f7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 42a9029a7aec461eb55f998a506b41f7; no voters:
I20260924 09:57:52.071043 11742 leader_election.cc:288] T 00000000000000000000000000000000 P 42a9029a7aec461eb55f998a506b41f7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 09:57:52.071640 11747 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 42a9029a7aec461eb55f998a506b41f7 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 09:57:52.072206 11747 raft_consensus.cc:686] T 00000000000000000000000000000000 P 42a9029a7aec461eb55f998a506b41f7 [term 1 LEADER]: Becoming Leader. State: Replica: 42a9029a7aec461eb55f998a506b41f7, State: Running, Role: LEADER
I20260924 09:57:52.072695 11747 consensus_queue.cc:237] T 00000000000000000000000000000000 P 42a9029a7aec461eb55f998a506b41f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42a9029a7aec461eb55f998a506b41f7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40811 } }
I20260924 09:57:52.073258 11742 sys_catalog.cc:564] T 00000000000000000000000000000000 P 42a9029a7aec461eb55f998a506b41f7 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 09:57:52.075141 11749 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42a9029a7aec461eb55f998a506b41f7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 42a9029a7aec461eb55f998a506b41f7. Latest consensus state: current_term: 1 leader_uuid: "42a9029a7aec461eb55f998a506b41f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42a9029a7aec461eb55f998a506b41f7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40811 } } }
I20260924 09:57:52.075541 11749 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42a9029a7aec461eb55f998a506b41f7 [sys.catalog]: This master's current role is: LEADER
I20260924 09:57:52.076244 11748 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42a9029a7aec461eb55f998a506b41f7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "42a9029a7aec461eb55f998a506b41f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42a9029a7aec461eb55f998a506b41f7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40811 } } }
I20260924 09:57:52.076562 11748 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42a9029a7aec461eb55f998a506b41f7 [sys.catalog]: This master's current role is: LEADER
I20260924 09:57:52.078688 11753 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 09:57:52.083884 11753 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 09:57:52.088413 11742 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790243735634835-388-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260924 09:57:52.099543 11753 catalog_manager.cc:1347] Generated new cluster ID: 6a7e8f94356942e2b519eb7683d5fba2
I20260924 09:57:52.099687 11753 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 09:57:52.116467 11753 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 09:57:52.118383 11753 catalog_manager.cc:1504] Loading token signing keys...
I20260924 09:57:52.132761 11753 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 42a9029a7aec461eb55f998a506b41f7: Generated new TSK 0
I20260924 09:57:52.133981 11753 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790243735634835-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790243735634835-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260924 09:57:52.839035 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
/tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790243735634835-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790243735634835-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790243735634835-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790243735634835-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:43821/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40811
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790243735634835-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790243735634835-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790243735634835-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790243735634835-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 09:57:53 dist-test-slave-w8d2 krb5kdc[11231](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790243873, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260924 09:57:53.230160 11787 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 09:57:53.230490 11787 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 09:57:53.230645 11787 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 09:57:53.237916 11787 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 09:57:53.238046 11787 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 09:57:53.238229 11787 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260924 09:57:53.248085 11787 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790243735634835-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790243735634835-388-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.97.20:43821/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790243735634835-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790243735634835-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40811
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790243735634835-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 09:57:53.249459 11787 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 09:57:53.250906 11787 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 09:57:53.266996 11797 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:57:53.273846 11798 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:57:53.289211 11800 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 09:57:53.289961 11787 server_base.cc:1034] running on GCE node
I20260924 09:57:53.290803 11787 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 09:57:53.292200 11787 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 09:57:53.293526 11787 hybrid_clock.cc:648] HybridClock initialized: now 1790243873293436 us; error 68 us; skew 500 ppm
I20260924 09:57:53.293963 11787 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 09:57:53 dist-test-slave-w8d2 krb5kdc[11231](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243873, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 09:57:53.298076 11787 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260924 09:57:53.304883 11787 webserver.cc:460] Webserver started at http://127.0.97.1:36669/ using document root <none> and password file <none>
I20260924 09:57:53.305833 11787 fs_manager.cc:359] Metadata directory not provided
I20260924 09:57:53.306001 11787 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 09:57:53.306376 11787 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 09:57:53 dist-test-slave-w8d2 krb5kdc[11231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243873, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260924 09:57:53.330214 11787 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790243735634835-388-0/minicluster-data/ts-0/data/instance:
uuid: "3f77db6ee4264d4c8ca1881c368b6f40"
format_stamp: "Formatted at 2026-09-24 09:57:53 on dist-test-slave-w8d2"
server_key: "d2c1ec1f1ffb64f8dd30210f45b7dfd4"
server_key_iv: "b7e4e344d8fcce0efc79b677de09529b"
server_key_version: "kuduclusterkey@0"
I20260924 09:57:53.331617 11787 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790243735634835-388-0/minicluster-data/ts-0/wal/instance:
uuid: "3f77db6ee4264d4c8ca1881c368b6f40"
format_stamp: "Formatted at 2026-09-24 09:57:53 on dist-test-slave-w8d2"
server_key: "d2c1ec1f1ffb64f8dd30210f45b7dfd4"
server_key_iv: "b7e4e344d8fcce0efc79b677de09529b"
server_key_version: "kuduclusterkey@0"
I20260924 09:57:53.339542 11787 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.000s
I20260924 09:57:53.370146 11807 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 09:57:53.372874 11787 fs_manager.cc:725] Time spent opening block manager: real 0.014s user 0.002s sys 0.001s
I20260924 09:57:53.373157 11787 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790243735634835-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790243735634835-388-0/minicluster-data/ts-0/wal
uuid: "3f77db6ee4264d4c8ca1881c368b6f40"
format_stamp: "Formatted at 2026-09-24 09:57:53 on dist-test-slave-w8d2"
server_key: "d2c1ec1f1ffb64f8dd30210f45b7dfd4"
server_key_iv: "b7e4e344d8fcce0efc79b677de09529b"
server_key_version: "kuduclusterkey@0"
I20260924 09:57:53.373519 11787 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790243735634835-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790243735634835-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790243735634835-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 09:57:53.409947 11787 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 09:57:53.416128 11787 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 09:57:53.416486 11787 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 09:57:53.417744 11787 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 09:57:53.419704 11787 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 09:57:53.419795 11787 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 09:57:53.423977 11787 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 09:57:53.424116 11787 ts_tablet_manager.cc:589] Time spent register tablets: real 0.004s user 0.000s sys 0.000s
Sep 24 09:57:53 dist-test-slave-w8d2 krb5kdc[11231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243873, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260924 09:57:53.955797 11787 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:42377
I20260924 09:57:53.955847 11919 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:42377 every 8 connection(s)
I20260924 09:57:53.957525 11787 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790243735634835-388-0/minicluster-data/ts-0/data/info.pb
I20260924 09:57:53.962918 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu as pid 11787
I20260924 09:57:53.963106 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790243735634835-388-0/minicluster-data/ts-0/wal/instance
I20260924 09:57:53.963920 388 external_mini_cluster.cc:1428] Setting key f8ebc63535d14ed2f71a0b256f9df5fe
Sep 24 09:57:53 dist-test-slave-w8d2 krb5kdc[11231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243873, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260924 09:57:53.995034 11922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:57:53.960976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:51631 (local address 127.0.97.62:40811)
0924 09:57:53.973485 (+ 12509us) server_negotiation.cc:206] Beginning negotiation
0924 09:57:53.973491 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 09:57:53.974369 (+ 878us) server_negotiation.cc:407] Connection header received
0924 09:57:53.977088 (+ 2719us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:57:53.977094 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:57:53.977192 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:57:53.977266 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:57:53.979637 (+ 2371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:57:53.980733 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:57:53.982947 (+ 2214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:57:53.983521 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:57:53.983607 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:57:53.988267 (+ 4660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:57:53.988274 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:57:53.988278 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:57:53.988316 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:57:53.991463 (+ 3147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:57:53.992360 (+ 897us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:57:53.992365 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:57:53.992369 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:57:53.992506 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:57:53.992998 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:57:53.993003 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:57:53.993005 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:57:53.993314 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:57:53.993422 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0924 09:57:53.994408 (+ 986us) server_negotiation.cc:299] Negotiation successful
0924 09:57:53.994653 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12387,"thread_start_us":12241,"threads_started":1}
I20260924 09:57:53.996174 11923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:57:53.961517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40811 (local address 127.0.97.1:51631)
0924 09:57:53.974190 (+ 12673us) negotiation.cc:107] Waiting for socket to connect
0924 09:57:53.974252 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0924 09:57:53.976729 (+ 2477us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:57:53.977895 (+ 1166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:57:53.977908 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:57:53.978518 (+ 610us) client_negotiation.cc:189] Negotiated authn=SASL
0924 09:57:53.979421 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:57:53.979447 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:57:53.980948 (+ 1501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:57:53.980953 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:57:53.982788 (+ 1835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:57:53.982807 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:57:53.983936 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:57:53.983943 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:57:53.984197 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:57:53.984982 (+ 785us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 09:57:53.985011 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0924 09:57:53.988008 (+ 2997us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 09:57:53.991651 (+ 3643us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 09:57:53.991661 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 09:57:53.991684 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0924 09:57:53.992230 (+ 546us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 09:57:53.992650 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 09:57:53.992655 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 09:57:53.992658 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 09:57:53.992884 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 09:57:53.993515 (+ 631us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 09:57:53.993523 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 09:57:53.993943 (+ 420us) client_negotiation.cc:769] Sending connection context
0924 09:57:53.994243 (+ 300us) client_negotiation.cc:240] Negotiation successful
0924 09:57:53.994768 (+ 525us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12252,"thread_start_us":12605,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260924 09:57:53.998224 11920 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40811
I20260924 09:57:53.998553 11920 heartbeater.cc:464] Registering TS with master...
I20260924 09:57:53.999715 11920 heartbeater.cc:511] Master 127.0.97.62:40811 requested a full tablet report, sending...
I20260924 09:57:54.002265 11707 ts_manager.cc:194] Registered new tserver with Master: 3f77db6ee4264d4c8ca1881c368b6f40 (127.0.97.1:42377)
I20260924 09:57:54.004298 11707 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:51631
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790243735634835-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790243735634835-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260924 09:57:54.034370 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
/tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790243735634835-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790243735634835-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790243735634835-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790243735634835-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:43821/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40811
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790243735634835-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790243735634835-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790243735634835-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790243735634835-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 09:57:54 dist-test-slave-w8d2 krb5kdc[11231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243873, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
W20260924 09:57:54.483763 11927 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 09:57:54.484201 11927 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 09:57:54.484365 11927 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 09:57:54.495851 11927 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 09:57:54.495997 11927 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 09:57:54.496215 11927 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260924 09:57:54.512838 11927 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790243735634835-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790243735634835-388-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.97.20:43821/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790243735634835-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790243735634835-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40811
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790243735634835-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 09:57:54.514590 11927 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 09:57:54.516395 11927 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 09:57:54.532260 11940 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:57:54.532635 11941 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:57:54.537061 11943 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 09:57:54.538945 11927 server_base.cc:1034] running on GCE node
I20260924 09:57:54.539696 11927 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 09:57:54.540921 11927 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 09:57:54.542229 11927 hybrid_clock.cc:648] HybridClock initialized: now 1790243874542125 us; error 85 us; skew 500 ppm
I20260924 09:57:54.542613 11927 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 09:57:54 dist-test-slave-w8d2 krb5kdc[11231](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243874, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 09:57:54.547118 11927 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260924 09:57:54.549115 11927 webserver.cc:460] Webserver started at http://127.0.97.2:45011/ using document root <none> and password file <none>
I20260924 09:57:54.550053 11927 fs_manager.cc:359] Metadata directory not provided
I20260924 09:57:54.550192 11927 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 09:57:54.550544 11927 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 09:57:54 dist-test-slave-w8d2 krb5kdc[11231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243874, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260924 09:57:54.568349 11927 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790243735634835-388-0/minicluster-data/ts-1/data/instance:
uuid: "3c32998a2e494252852add6da4b30de3"
format_stamp: "Formatted at 2026-09-24 09:57:54 on dist-test-slave-w8d2"
server_key: "5310e2a0e5935d794ed806df003cd208"
server_key_iv: "673fb83e76cc3a9963d41103b6693f49"
server_key_version: "kuduclusterkey@0"
I20260924 09:57:54.569765 11927 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790243735634835-388-0/minicluster-data/ts-1/wal/instance:
uuid: "3c32998a2e494252852add6da4b30de3"
format_stamp: "Formatted at 2026-09-24 09:57:54 on dist-test-slave-w8d2"
server_key: "5310e2a0e5935d794ed806df003cd208"
server_key_iv: "673fb83e76cc3a9963d41103b6693f49"
server_key_version: "kuduclusterkey@0"
I20260924 09:57:54.576857 11927 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.000s sys 0.006s
I20260924 09:57:54.595801 11950 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 09:57:54.597312 11927 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.001s sys 0.003s
I20260924 09:57:54.597486 11927 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790243735634835-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790243735634835-388-0/minicluster-data/ts-1/wal
uuid: "3c32998a2e494252852add6da4b30de3"
format_stamp: "Formatted at 2026-09-24 09:57:54 on dist-test-slave-w8d2"
server_key: "5310e2a0e5935d794ed806df003cd208"
server_key_iv: "673fb83e76cc3a9963d41103b6693f49"
server_key_version: "kuduclusterkey@0"
I20260924 09:57:54.597704 11927 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790243735634835-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790243735634835-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790243735634835-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 09:57:54.640142 11927 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 09:57:54.644136 11927 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 09:57:54.644416 11927 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 09:57:54.645836 11927 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 09:57:54.647209 11927 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 09:57:54.647312 11927 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 09:57:54.647437 11927 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 09:57:54.647490 11927 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 09:57:54.681937 11927 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:43003
I20260924 09:57:54.681957 12062 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:43003 every 8 connection(s)
I20260924 09:57:54.683193 11927 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790243735634835-388-0/minicluster-data/ts-1/data/info.pb
I20260924 09:57:54.685431 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu as pid 11927
I20260924 09:57:54.685595 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790243735634835-388-0/minicluster-data/ts-1/wal/instance
I20260924 09:57:54.686133 388 external_mini_cluster.cc:1428] Setting key 793ac88acfb9775364f22cf52a16f822
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Sep 24 09:57:54 dist-test-slave-w8d2 krb5kdc[11231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243874, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 09:57:54.720065 12065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:57:54.685686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:53463 (local address 127.0.97.62:40811)
0924 09:57:54.701280 (+ 15594us) server_negotiation.cc:206] Beginning negotiation
0924 09:57:54.701289 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 09:57:54.706509 (+ 5220us) server_negotiation.cc:407] Connection header received
0924 09:57:54.706570 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:57:54.706573 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:57:54.706656 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:57:54.706755 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:57:54.708418 (+ 1663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:57:54.709152 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:57:54.710694 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:57:54.711099 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:57:54.711201 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:57:54.714700 (+ 3499us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:57:54.714706 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:57:54.714709 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:57:54.714739 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:57:54.717216 (+ 2477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:57:54.717881 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:57:54.717884 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:57:54.717887 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:57:54.718005 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:57:54.718409 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:57:54.718412 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:57:54.718415 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:57:54.718682 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:57:54.718805 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0924 09:57:54.719493 (+ 688us) server_negotiation.cc:299] Negotiation successful
0924 09:57:54.719707 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15434,"thread_start_us":15602,"threads_started":1}
I20260924 09:57:54.721067 12066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:57:54.686356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40811 (local address 127.0.97.2:53463)
0924 09:57:54.703863 (+ 17507us) negotiation.cc:107] Waiting for socket to connect
0924 09:57:54.703934 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0924 09:57:54.706168 (+ 2234us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:57:54.707136 (+ 968us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:57:54.707145 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:57:54.707607 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0924 09:57:54.708233 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:57:54.708251 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:57:54.709306 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:57:54.709309 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:57:54.710565 (+ 1256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:57:54.710576 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:57:54.711343 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:57:54.711350 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:57:54.711572 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:57:54.712207 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 09:57:54.712230 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 09:57:54.714491 (+ 2261us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 09:57:54.717347 (+ 2856us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 09:57:54.717356 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 09:57:54.717373 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 09:57:54.717765 (+ 392us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 09:57:54.718100 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 09:57:54.718105 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 09:57:54.718108 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 09:57:54.718299 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 09:57:54.718801 (+ 502us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 09:57:54.718809 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 09:57:54.719118 (+ 309us) client_negotiation.cc:769] Sending connection context
0924 09:57:54.719377 (+ 259us) client_negotiation.cc:240] Negotiation successful
0924 09:57:54.719867 (+ 490us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17185,"thread_start_us":16944,"threads_started":1}
I20260924 09:57:54.722657 12063 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40811
I20260924 09:57:54.722997 12063 heartbeater.cc:464] Registering TS with master...
I20260924 09:57:54.723901 12063 heartbeater.cc:511] Master 127.0.97.62:40811 requested a full tablet report, sending...
I20260924 09:57:54.725566 11707 ts_manager.cc:194] Registered new tserver with Master: 3c32998a2e494252852add6da4b30de3 (127.0.97.2:43003)
I20260924 09:57:54.726606 11707 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:53463
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790243735634835-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790243735634835-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260924 09:57:54.739693 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
/tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790243735634835-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790243735634835-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790243735634835-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790243735634835-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:43821/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40811
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790243735634835-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790243735634835-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790243735634835-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790243735634835-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 09:57:55.008177 11920 heartbeater.cc:503] Master 127.0.97.62:40811 was elected leader, sending a full tablet report...
W20260924 09:57:55.059190 12069 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 09:57:55.059497 12069 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 09:57:55.059635 12069 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 09:57:55.066731 12069 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 09:57:55.066854 12069 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 09:57:55.067009 12069 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260924 09:57:55.076951 12069 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790243735634835-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790243735634835-388-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.97.20:43821/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790243735634835-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790243735634835-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40811
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790243735634835-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 09:57:55.078256 12069 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 09:57:55.080026 12069 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 09:57:55.092768 12074 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:57:55.092898 12077 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:57:55.092824 12075 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 09:57:55.093936 12069 server_base.cc:1034] running on GCE node
I20260924 09:57:55.094519 12069 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 09:57:55.095417 12069 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 09:57:55.096617 12069 hybrid_clock.cc:648] HybridClock initialized: now 1790243875096573 us; error 35 us; skew 500 ppm
I20260924 09:57:55.096905 12069 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 09:57:55 dist-test-slave-w8d2 krb5kdc[11231](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243875, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 09:57:55.099920 12069 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260924 09:57:55.101261 12069 webserver.cc:460] Webserver started at http://127.0.97.3:38633/ using document root <none> and password file <none>
I20260924 09:57:55.101919 12069 fs_manager.cc:359] Metadata directory not provided
I20260924 09:57:55.102029 12069 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 09:57:55.102310 12069 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 09:57:55 dist-test-slave-w8d2 krb5kdc[11231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243875, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260924 09:57:55.118260 12069 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790243735634835-388-0/minicluster-data/ts-2/data/instance:
uuid: "6951a644221047f9b7653878dd24b922"
format_stamp: "Formatted at 2026-09-24 09:57:55 on dist-test-slave-w8d2"
server_key: "7777acbbb04118f6ca99528516b1ed3c"
server_key_iv: "93253d0b6ced1eaa8c0aca592b3b271d"
server_key_version: "kuduclusterkey@0"
I20260924 09:57:55.119390 12069 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790243735634835-388-0/minicluster-data/ts-2/wal/instance:
uuid: "6951a644221047f9b7653878dd24b922"
format_stamp: "Formatted at 2026-09-24 09:57:55 on dist-test-slave-w8d2"
server_key: "7777acbbb04118f6ca99528516b1ed3c"
server_key_iv: "93253d0b6ced1eaa8c0aca592b3b271d"
server_key_version: "kuduclusterkey@0"
I20260924 09:57:55.126035 12069 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260924 09:57:55.142596 12084 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 09:57:55.144362 12069 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260924 09:57:55.144553 12069 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790243735634835-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790243735634835-388-0/minicluster-data/ts-2/wal
uuid: "6951a644221047f9b7653878dd24b922"
format_stamp: "Formatted at 2026-09-24 09:57:55 on dist-test-slave-w8d2"
server_key: "7777acbbb04118f6ca99528516b1ed3c"
server_key_iv: "93253d0b6ced1eaa8c0aca592b3b271d"
server_key_version: "kuduclusterkey@0"
I20260924 09:57:55.144809 12069 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790243735634835-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790243735634835-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790243735634835-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 09:57:55.164726 12069 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 09:57:55.169418 12069 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 09:57:55.169723 12069 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 09:57:55.171337 12069 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 09:57:55.173002 12069 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 09:57:55.173087 12069 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 09:57:55.173197 12069 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 09:57:55.173254 12069 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 09:57:55.205726 12069 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:39893
I20260924 09:57:55.205752 12196 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:39893 every 8 connection(s)
I20260924 09:57:55.207018 12069 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790243735634835-388-0/minicluster-data/ts-2/data/info.pb
I20260924 09:57:55.212152 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu as pid 12069
I20260924 09:57:55.212327 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790243735634835-388-0/minicluster-data/ts-2/wal/instance
I20260924 09:57:55.212852 388 external_mini_cluster.cc:1428] Setting key 5d5d86919a6b32dce0b378af3c9bc716
Sep 24 09:57:55 dist-test-slave-w8d2 krb5kdc[11231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243875, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 09:57:55.226016 12065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:57:55.209473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:42103 (local address 127.0.97.62:40811)
0924 09:57:55.209637 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0924 09:57:55.209642 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 09:57:55.210699 (+ 1057us) server_negotiation.cc:407] Connection header received
0924 09:57:55.212631 (+ 1932us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:57:55.212636 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:57:55.212717 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:57:55.212810 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:57:55.214940 (+ 2130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:57:55.215762 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:57:55.217315 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:57:55.217720 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:57:55.217819 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:57:55.220978 (+ 3159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:57:55.220982 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:57:55.220986 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:57:55.221016 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:57:55.223437 (+ 2421us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:57:55.224118 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:57:55.224122 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:57:55.224124 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:57:55.224254 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:57:55.224641 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:57:55.224643 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:57:55.224645 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:57:55.224917 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:57:55.225021 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0924 09:57:55.225615 (+ 594us) server_negotiation.cc:299] Negotiation successful
0924 09:57:55.225782 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 09:57:55.227038 12199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:57:55.209882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40811 (local address 127.0.97.3:42103)
0924 09:57:55.210545 (+ 663us) negotiation.cc:107] Waiting for socket to connect
0924 09:57:55.210582 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 09:57:55.212330 (+ 1748us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:57:55.213091 (+ 761us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:57:55.213103 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:57:55.214104 (+ 1001us) client_negotiation.cc:189] Negotiated authn=SASL
0924 09:57:55.214792 (+ 688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:57:55.214811 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:57:55.215914 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:57:55.215920 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:57:55.217193 (+ 1273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:57:55.217204 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:57:55.217914 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:57:55.217919 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:57:55.218195 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:57:55.218791 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 09:57:55.218813 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 09:57:55.220782 (+ 1969us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 09:57:55.223559 (+ 2777us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 09:57:55.223567 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 09:57:55.223584 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 09:57:55.223994 (+ 410us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 09:57:55.224358 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 09:57:55.224361 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 09:57:55.224364 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 09:57:55.224550 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 09:57:55.225010 (+ 460us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 09:57:55.225015 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 09:57:55.225304 (+ 289us) client_negotiation.cc:769] Sending connection context
0924 09:57:55.225525 (+ 221us) client_negotiation.cc:240] Negotiation successful
0924 09:57:55.225879 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":278,"threads_started":1}
I20260924 09:57:55.228434 12197 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40811
I20260924 09:57:55.228683 12197 heartbeater.cc:464] Registering TS with master...
I20260924 09:57:55.229339 12197 heartbeater.cc:511] Master 127.0.97.62:40811 requested a full tablet report, sending...
I20260924 09:57:55.230660 11706 ts_manager.cc:194] Registered new tserver with Master: 6951a644221047f9b7653878dd24b922 (127.0.97.3:39893)
I20260924 09:57:55.231534 11706 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:42103
I20260924 09:57:55.239965 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 09:57:55.255888 12065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:57:55.244070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35246 (local address 127.0.97.62:40811)
0924 09:57:55.244256 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0924 09:57:55.244263 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 09:57:55.244582 (+ 319us) server_negotiation.cc:407] Connection header received
0924 09:57:55.244701 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:57:55.244705 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:57:55.244773 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:57:55.244846 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:57:55.246557 (+ 1711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:57:55.247287 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:57:55.248458 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:57:55.248857 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:57:55.248945 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:57:55.250992 (+ 2047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:57:55.250997 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:57:55.251000 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:57:55.251035 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:57:55.253611 (+ 2576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:57:55.254154 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:57:55.254158 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:57:55.254160 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:57:55.254273 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:57:55.254566 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:57:55.254570 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:57:55.254572 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:57:55.254831 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:57:55.255025 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0924 09:57:55.255334 (+ 309us) server_negotiation.cc:299] Negotiation successful
0924 09:57:55.255590 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 09:57:55.259629 11706 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35246:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 09:57:55.266793 11706 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 09:57:55.290385 12209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:57:55.278943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:43003 (local address 127.0.0.1:44828)
0924 09:57:55.279954 (+ 1011us) negotiation.cc:107] Waiting for socket to connect
0924 09:57:55.279995 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 09:57:55.280168 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:57:55.281363 (+ 1195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:57:55.281372 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:57:55.281405 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 09:57:55.281866 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:57:55.281879 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:57:55.284032 (+ 2153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:57:55.284038 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:57:55.287109 (+ 3071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:57:55.287125 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:57:55.289333 (+ 2208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:57:55.289339 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:57:55.289571 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:57:55.289625 (+ 54us) client_negotiation.cc:769] Sending connection context
0924 09:57:55.289754 (+ 129us) client_negotiation.cc:240] Negotiation successful
0924 09:57:55.290006 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":751,"thread_start_us":611,"threads_started":1}
I20260924 09:57:55.292047 12211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:57:55.279396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44828 (local address 127.0.97.2:43003)
0924 09:57:55.280731 (+ 1335us) server_negotiation.cc:206] Beginning negotiation
0924 09:57:55.280740 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 09:57:55.280768 (+ 28us) server_negotiation.cc:407] Connection header received
0924 09:57:55.280864 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:57:55.280872 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:57:55.281126 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:57:55.282062 (+ 936us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 09:57:55.282268 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:57:55.283873 (+ 1605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:57:55.287337 (+ 3464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:57:55.288851 (+ 1514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:57:55.288974 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:57:55.289184 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0924 09:57:55.291526 (+ 2342us) server_negotiation.cc:299] Negotiation successful
0924 09:57:55.291716 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1144,"thread_start_us":1047,"threads_started":1}
I20260924 09:57:55.296379 11998 tablet_service.cc:1467] Processing CreateTablet for tablet b74539682c2c4e6e9fe210fafc7ca76c (DEFAULT_TABLE table=test-table [id=228585f5e5954a3f912c3df36c4a976c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 09:57:55.298056 11998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b74539682c2c4e6e9fe210fafc7ca76c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 09:57:55.300881 12209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:57:55.278980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42377 (local address 127.0.0.1:34044)
0924 09:57:55.291198 (+ 12218us) negotiation.cc:107] Waiting for socket to connect
0924 09:57:55.291218 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 09:57:55.291346 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:57:55.293943 (+ 2597us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:57:55.293949 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:57:55.293975 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 09:57:55.294150 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:57:55.294159 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:57:55.296528 (+ 2369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:57:55.296534 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:57:55.298565 (+ 2031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:57:55.298580 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:57:55.300135 (+ 1555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:57:55.300142 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:57:55.300328 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:57:55.300361 (+ 33us) client_negotiation.cc:769] Sending connection context
0924 09:57:55.300452 (+ 91us) client_negotiation.cc:240] Negotiation successful
0924 09:57:55.300611 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12096,"thread_start_us":19900,"threads_started":1}
I20260924 09:57:55.301623 12212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:57:55.279805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34044 (local address 127.0.97.1:42377)
0924 09:57:55.289813 (+ 10008us) server_negotiation.cc:206] Beginning negotiation
0924 09:57:55.289833 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0924 09:57:55.293342 (+ 3509us) server_negotiation.cc:407] Connection header received
0924 09:57:55.293469 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:57:55.293478 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:57:55.293737 (+ 259us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:57:55.294314 (+ 577us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 09:57:55.294579 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:57:55.296340 (+ 1761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:57:55.298784 (+ 2444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:57:55.299828 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:57:55.299924 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:57:55.300132 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0924 09:57:55.301105 (+ 973us) server_negotiation.cc:299] Negotiation successful
0924 09:57:55.301290 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9844,"thread_start_us":9705,"threads_started":1}
I20260924 09:57:55.304813 11855 tablet_service.cc:1467] Processing CreateTablet for tablet b74539682c2c4e6e9fe210fafc7ca76c (DEFAULT_TABLE table=test-table [id=228585f5e5954a3f912c3df36c4a976c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 09:57:55.306052 11855 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b74539682c2c4e6e9fe210fafc7ca76c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 09:57:55.307677 12213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:57:55.280194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:39893 (local address 127.0.0.1:54186)
0924 09:57:55.298397 (+ 18203us) negotiation.cc:107] Waiting for socket to connect
0924 09:57:55.298415 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 09:57:55.298533 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:57:55.300748 (+ 2215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:57:55.300753 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:57:55.300773 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 09:57:55.301104 (+ 331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:57:55.301116 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:57:55.303377 (+ 2261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:57:55.303382 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:57:55.305234 (+ 1852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:57:55.305248 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:57:55.306913 (+ 1665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:57:55.306919 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:57:55.307075 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:57:55.307107 (+ 32us) client_negotiation.cc:769] Sending connection context
0924 09:57:55.307173 (+ 66us) client_negotiation.cc:240] Negotiation successful
0924 09:57:55.307357 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18080,"thread_start_us":18408,"threads_started":1}
I20260924 09:57:55.308452 12214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:57:55.280798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54186 (local address 127.0.97.3:39893)
0924 09:57:55.292514 (+ 11716us) server_negotiation.cc:206] Beginning negotiation
0924 09:57:55.292523 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 09:57:55.300204 (+ 7681us) server_negotiation.cc:407] Connection header received
0924 09:57:55.300301 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:57:55.300309 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:57:55.300562 (+ 253us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:57:55.301300 (+ 738us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 09:57:55.301536 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:57:55.303226 (+ 1690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:57:55.305418 (+ 2192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:57:55.306708 (+ 1290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:57:55.307612 (+ 904us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:57:55.307909 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0924 09:57:55.307975 (+ 66us) server_negotiation.cc:299] Negotiation successful
0924 09:57:55.308117 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11544,"thread_start_us":11719,"threads_started":1}
I20260924 09:57:55.311405 12215 tablet_bootstrap.cc:492] T b74539682c2c4e6e9fe210fafc7ca76c P 3c32998a2e494252852add6da4b30de3: Bootstrap starting.
I20260924 09:57:55.311939 12132 tablet_service.cc:1467] Processing CreateTablet for tablet b74539682c2c4e6e9fe210fafc7ca76c (DEFAULT_TABLE table=test-table [id=228585f5e5954a3f912c3df36c4a976c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 09:57:55.313277 12132 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b74539682c2c4e6e9fe210fafc7ca76c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 09:57:55.314759 12215 tablet_bootstrap.cc:654] T b74539682c2c4e6e9fe210fafc7ca76c P 3c32998a2e494252852add6da4b30de3: Neither blocks nor log segments found. Creating new log.
I20260924 09:57:55.315954 12215 log.cc:824] T b74539682c2c4e6e9fe210fafc7ca76c P 3c32998a2e494252852add6da4b30de3: Log is configured to *not* fsync() on all Append() calls
I20260924 09:57:55.318930 12215 tablet_bootstrap.cc:492] T b74539682c2c4e6e9fe210fafc7ca76c P 3c32998a2e494252852add6da4b30de3: No bootstrap required, opened a new log
I20260924 09:57:55.319080 12217 tablet_bootstrap.cc:492] T b74539682c2c4e6e9fe210fafc7ca76c P 3f77db6ee4264d4c8ca1881c368b6f40: Bootstrap starting.
I20260924 09:57:55.319283 12215 ts_tablet_manager.cc:1397] T b74539682c2c4e6e9fe210fafc7ca76c P 3c32998a2e494252852add6da4b30de3: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260924 09:57:55.322680 12217 tablet_bootstrap.cc:654] T b74539682c2c4e6e9fe210fafc7ca76c P 3f77db6ee4264d4c8ca1881c368b6f40: Neither blocks nor log segments found. Creating new log.
I20260924 09:57:55.323899 12217 log.cc:824] T b74539682c2c4e6e9fe210fafc7ca76c P 3f77db6ee4264d4c8ca1881c368b6f40: Log is configured to *not* fsync() on all Append() calls
I20260924 09:57:55.327158 12217 tablet_bootstrap.cc:492] T b74539682c2c4e6e9fe210fafc7ca76c P 3f77db6ee4264d4c8ca1881c368b6f40: No bootstrap required, opened a new log
I20260924 09:57:55.327476 12217 ts_tablet_manager.cc:1397] T b74539682c2c4e6e9fe210fafc7ca76c P 3f77db6ee4264d4c8ca1881c368b6f40: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260924 09:57:55.327280 12215 raft_consensus.cc:348] T b74539682c2c4e6e9fe210fafc7ca76c P 3c32998a2e494252852add6da4b30de3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c32998a2e494252852add6da4b30de3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43003 } } peers { permanent_uuid: "3f77db6ee4264d4c8ca1881c368b6f40" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42377 } } peers { permanent_uuid: "6951a644221047f9b7653878dd24b922" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39893 } }
I20260924 09:57:55.327693 12215 raft_consensus.cc:374] T b74539682c2c4e6e9fe210fafc7ca76c P 3c32998a2e494252852add6da4b30de3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 09:57:55.327713 12219 tablet_bootstrap.cc:492] T b74539682c2c4e6e9fe210fafc7ca76c P 6951a644221047f9b7653878dd24b922: Bootstrap starting.
I20260924 09:57:55.327783 12215 raft_consensus.cc:729] T b74539682c2c4e6e9fe210fafc7ca76c P 3c32998a2e494252852add6da4b30de3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c32998a2e494252852add6da4b30de3, State: Initialized, Role: FOLLOWER
I20260924 09:57:55.328795 12215 consensus_queue.cc:260] T b74539682c2c4e6e9fe210fafc7ca76c P 3c32998a2e494252852add6da4b30de3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c32998a2e494252852add6da4b30de3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43003 } } peers { permanent_uuid: "3f77db6ee4264d4c8ca1881c368b6f40" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42377 } } peers { permanent_uuid: "6951a644221047f9b7653878dd24b922" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39893 } }
I20260924 09:57:55.329859 12063 heartbeater.cc:503] Master 127.0.97.62:40811 was elected leader, sending a full tablet report...
I20260924 09:57:55.330494 12215 ts_tablet_manager.cc:1428] T b74539682c2c4e6e9fe210fafc7ca76c P 3c32998a2e494252852add6da4b30de3: Time spent starting tablet: real 0.011s user 0.006s sys 0.003s
I20260924 09:57:55.331061 12219 tablet_bootstrap.cc:654] T b74539682c2c4e6e9fe210fafc7ca76c P 6951a644221047f9b7653878dd24b922: Neither blocks nor log segments found. Creating new log.
I20260924 09:57:55.332129 12219 log.cc:824] T b74539682c2c4e6e9fe210fafc7ca76c P 6951a644221047f9b7653878dd24b922: Log is configured to *not* fsync() on all Append() calls
I20260924 09:57:55.335817 12219 tablet_bootstrap.cc:492] T b74539682c2c4e6e9fe210fafc7ca76c P 6951a644221047f9b7653878dd24b922: No bootstrap required, opened a new log
I20260924 09:57:55.336076 12219 ts_tablet_manager.cc:1397] T b74539682c2c4e6e9fe210fafc7ca76c P 6951a644221047f9b7653878dd24b922: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260924 09:57:55.336949 12217 raft_consensus.cc:348] T b74539682c2c4e6e9fe210fafc7ca76c P 3f77db6ee4264d4c8ca1881c368b6f40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c32998a2e494252852add6da4b30de3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43003 } } peers { permanent_uuid: "3f77db6ee4264d4c8ca1881c368b6f40" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42377 } } peers { permanent_uuid: "6951a644221047f9b7653878dd24b922" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39893 } }
I20260924 09:57:55.337384 12217 raft_consensus.cc:374] T b74539682c2c4e6e9fe210fafc7ca76c P 3f77db6ee4264d4c8ca1881c368b6f40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 09:57:55.337488 12217 raft_consensus.cc:729] T b74539682c2c4e6e9fe210fafc7ca76c P 3f77db6ee4264d4c8ca1881c368b6f40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f77db6ee4264d4c8ca1881c368b6f40, State: Initialized, Role: FOLLOWER
I20260924 09:57:55.338074 12217 consensus_queue.cc:260] T b74539682c2c4e6e9fe210fafc7ca76c P 3f77db6ee4264d4c8ca1881c368b6f40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c32998a2e494252852add6da4b30de3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43003 } } peers { permanent_uuid: "3f77db6ee4264d4c8ca1881c368b6f40" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42377 } } peers { permanent_uuid: "6951a644221047f9b7653878dd24b922" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39893 } }
I20260924 09:57:55.339725 12217 ts_tablet_manager.cc:1428] T b74539682c2c4e6e9fe210fafc7ca76c P 3f77db6ee4264d4c8ca1881c368b6f40: Time spent starting tablet: real 0.012s user 0.007s sys 0.005s
I20260924 09:57:55.344463 12219 raft_consensus.cc:348] T b74539682c2c4e6e9fe210fafc7ca76c P 6951a644221047f9b7653878dd24b922 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c32998a2e494252852add6da4b30de3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43003 } } peers { permanent_uuid: "3f77db6ee4264d4c8ca1881c368b6f40" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42377 } } peers { permanent_uuid: "6951a644221047f9b7653878dd24b922" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39893 } }
I20260924 09:57:55.344802 12219 raft_consensus.cc:374] T b74539682c2c4e6e9fe210fafc7ca76c P 6951a644221047f9b7653878dd24b922 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 09:57:55.344899 12219 raft_consensus.cc:729] T b74539682c2c4e6e9fe210fafc7ca76c P 6951a644221047f9b7653878dd24b922 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6951a644221047f9b7653878dd24b922, State: Initialized, Role: FOLLOWER
I20260924 09:57:55.345489 12219 consensus_queue.cc:260] T b74539682c2c4e6e9fe210fafc7ca76c P 6951a644221047f9b7653878dd24b922 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c32998a2e494252852add6da4b30de3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43003 } } peers { permanent_uuid: "3f77db6ee4264d4c8ca1881c368b6f40" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42377 } } peers { permanent_uuid: "6951a644221047f9b7653878dd24b922" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39893 } }
I20260924 09:57:55.346472 12197 heartbeater.cc:503] Master 127.0.97.62:40811 was elected leader, sending a full tablet report...
I20260924 09:57:55.346947 12219 ts_tablet_manager.cc:1428] T b74539682c2c4e6e9fe210fafc7ca76c P 6951a644221047f9b7653878dd24b922: Time spent starting tablet: real 0.011s user 0.011s sys 0.000s
I20260924 09:57:55.372316 12222 raft_consensus.cc:482] T b74539682c2c4e6e9fe210fafc7ca76c P 3f77db6ee4264d4c8ca1881c368b6f40 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 09:57:55.372596 12222 raft_consensus.cc:504] T b74539682c2c4e6e9fe210fafc7ca76c P 3f77db6ee4264d4c8ca1881c368b6f40 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c32998a2e494252852add6da4b30de3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43003 } } peers { permanent_uuid: "3f77db6ee4264d4c8ca1881c368b6f40" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42377 } } peers { permanent_uuid: "6951a644221047f9b7653878dd24b922" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39893 } }
I20260924 09:57:55.374320 12222 leader_election.cc:288] T b74539682c2c4e6e9fe210fafc7ca76c P 3f77db6ee4264d4c8ca1881c368b6f40 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c32998a2e494252852add6da4b30de3 (127.0.97.2:43003), 6951a644221047f9b7653878dd24b922 (127.0.97.3:39893)
I20260924 09:57:55.382871 12211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:57:55.374787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:34111 (local address 127.0.97.2:43003)
0924 09:57:55.375307 (+ 520us) server_negotiation.cc:206] Beginning negotiation
0924 09:57:55.375314 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 09:57:55.375728 (+ 414us) server_negotiation.cc:407] Connection header received
0924 09:57:55.375832 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:57:55.375837 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:57:55.375903 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:57:55.376000 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 09:57:55.376601 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:57:55.377600 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:57:55.381037 (+ 3437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:57:55.381981 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:57:55.382072 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:57:55.382135 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 09:57:55.382505 (+ 370us) server_negotiation.cc:299] Negotiation successful
0924 09:57:55.382629 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":349}
I20260924 09:57:55.382871 12214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:57:55.374896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:59797 (local address 127.0.97.3:39893)
0924 09:57:55.375103 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0924 09:57:55.375109 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 09:57:55.375476 (+ 367us) server_negotiation.cc:407] Connection header received
0924 09:57:55.375690 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:57:55.375695 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:57:55.375771 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:57:55.375856 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 09:57:55.376492 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:57:55.377600 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:57:55.380808 (+ 3208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:57:55.381791 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:57:55.381866 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:57:55.381939 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 09:57:55.382511 (+ 572us) server_negotiation.cc:299] Negotiation successful
0924 09:57:55.382629 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260924 09:57:55.383078 12224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:57:55.374664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:39893 (local address 127.0.97.1:59797)
0924 09:57:55.375363 (+ 699us) negotiation.cc:107] Waiting for socket to connect
0924 09:57:55.375382 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 09:57:55.375546 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:57:55.375880 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:57:55.375883 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:57:55.375904 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 09:57:55.376343 (+ 439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:57:55.376359 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:57:55.377725 (+ 1366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:57:55.377731 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:57:55.380634 (+ 2903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:57:55.380649 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:57:55.382098 (+ 1449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:57:55.382105 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:57:55.382328 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:57:55.382384 (+ 56us) client_negotiation.cc:769] Sending connection context
0924 09:57:55.382471 (+ 87us) client_negotiation.cc:240] Negotiation successful
0924 09:57:55.382646 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":528,"mutex_wait_us":174,"thread_start_us":352,"threads_started":1}
I20260924 09:57:55.383100 12225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:57:55.374682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:43003 (local address 127.0.97.1:34111)
0924 09:57:55.375639 (+ 957us) negotiation.cc:107] Waiting for socket to connect
0924 09:57:55.375654 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 09:57:55.375743 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:57:55.376028 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:57:55.376032 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:57:55.376051 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 09:57:55.376454 (+ 403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:57:55.376464 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:57:55.377725 (+ 1261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:57:55.377731 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:57:55.380906 (+ 3175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:57:55.380919 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:57:55.382168 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:57:55.382174 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:57:55.382335 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:57:55.382379 (+ 44us) client_negotiation.cc:769] Sending connection context
0924 09:57:55.382471 (+ 92us) client_negotiation.cc:240] Negotiation successful
0924 09:57:55.382647 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":853,"mutex_wait_us":190,"thread_start_us":699,"threads_started":1}
I20260924 09:57:55.384147 12018 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b74539682c2c4e6e9fe210fafc7ca76c" candidate_uuid: "3f77db6ee4264d4c8ca1881c368b6f40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c32998a2e494252852add6da4b30de3" is_pre_election: true
I20260924 09:57:55.384152 12152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b74539682c2c4e6e9fe210fafc7ca76c" candidate_uuid: "3f77db6ee4264d4c8ca1881c368b6f40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6951a644221047f9b7653878dd24b922" is_pre_election: true
I20260924 09:57:55.384591 12018 raft_consensus.cc:2454] T b74539682c2c4e6e9fe210fafc7ca76c P 3c32998a2e494252852add6da4b30de3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f77db6ee4264d4c8ca1881c368b6f40 in term 0.
I20260924 09:57:55.384593 12152 raft_consensus.cc:2454] T b74539682c2c4e6e9fe210fafc7ca76c P 6951a644221047f9b7653878dd24b922 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f77db6ee4264d4c8ca1881c368b6f40 in term 0.
I20260924 09:57:55.385152 11811 leader_election.cc:302] T b74539682c2c4e6e9fe210fafc7ca76c P 3f77db6ee4264d4c8ca1881c368b6f40 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c32998a2e494252852add6da4b30de3, 3f77db6ee4264d4c8ca1881c368b6f40; no voters:
I20260924 09:57:55.385493 12222 raft_consensus.cc:2781] T b74539682c2c4e6e9fe210fafc7ca76c P 3f77db6ee4264d4c8ca1881c368b6f40 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 09:57:55.385623 12222 raft_consensus.cc:482] T b74539682c2c4e6e9fe210fafc7ca76c P 3f77db6ee4264d4c8ca1881c368b6f40 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 09:57:55.385711 12222 raft_consensus.cc:3037] T b74539682c2c4e6e9fe210fafc7ca76c P 3f77db6ee4264d4c8ca1881c368b6f40 [term 0 FOLLOWER]: Advancing to term 1
I20260924 09:57:55.387789 12222 raft_consensus.cc:504] T b74539682c2c4e6e9fe210fafc7ca76c P 3f77db6ee4264d4c8ca1881c368b6f40 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c32998a2e494252852add6da4b30de3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43003 } } peers { permanent_uuid: "3f77db6ee4264d4c8ca1881c368b6f40" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42377 } } peers { permanent_uuid: "6951a644221047f9b7653878dd24b922" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39893 } }
I20260924 09:57:55.388324 12222 leader_election.cc:288] T b74539682c2c4e6e9fe210fafc7ca76c P 3f77db6ee4264d4c8ca1881c368b6f40 [CANDIDATE]: Term 1 election: Requested vote from peers 3c32998a2e494252852add6da4b30de3 (127.0.97.2:43003), 6951a644221047f9b7653878dd24b922 (127.0.97.3:39893)
I20260924 09:57:55.388716 12018 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b74539682c2c4e6e9fe210fafc7ca76c" candidate_uuid: "3f77db6ee4264d4c8ca1881c368b6f40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c32998a2e494252852add6da4b30de3"
I20260924 09:57:55.388784 12152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b74539682c2c4e6e9fe210fafc7ca76c" candidate_uuid: "3f77db6ee4264d4c8ca1881c368b6f40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6951a644221047f9b7653878dd24b922"
I20260924 09:57:55.388919 12018 raft_consensus.cc:3037] T b74539682c2c4e6e9fe210fafc7ca76c P 3c32998a2e494252852add6da4b30de3 [term 0 FOLLOWER]: Advancing to term 1
I20260924 09:57:55.388958 12152 raft_consensus.cc:3037] T b74539682c2c4e6e9fe210fafc7ca76c P 6951a644221047f9b7653878dd24b922 [term 0 FOLLOWER]: Advancing to term 1
I20260924 09:57:55.392554 12152 raft_consensus.cc:2454] T b74539682c2c4e6e9fe210fafc7ca76c P 6951a644221047f9b7653878dd24b922 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f77db6ee4264d4c8ca1881c368b6f40 in term 1.
I20260924 09:57:55.392554 12018 raft_consensus.cc:2454] T b74539682c2c4e6e9fe210fafc7ca76c P 3c32998a2e494252852add6da4b30de3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f77db6ee4264d4c8ca1881c368b6f40 in term 1.
I20260924 09:57:55.392972 11809 leader_election.cc:302] T b74539682c2c4e6e9fe210fafc7ca76c P 3f77db6ee4264d4c8ca1881c368b6f40 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 3c32998a2e494252852add6da4b30de3, 3f77db6ee4264d4c8ca1881c368b6f40, 6951a644221047f9b7653878dd24b922; no voters:
I20260924 09:57:55.393287 12222 raft_consensus.cc:2781] T b74539682c2c4e6e9fe210fafc7ca76c P 3f77db6ee4264d4c8ca1881c368b6f40 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 09:57:55.393690 12222 raft_consensus.cc:686] T b74539682c2c4e6e9fe210fafc7ca76c P 3f77db6ee4264d4c8ca1881c368b6f40 [term 1 LEADER]: Becoming Leader. State: Replica: 3f77db6ee4264d4c8ca1881c368b6f40, State: Running, Role: LEADER
I20260924 09:57:55.394121 12222 consensus_queue.cc:237] T b74539682c2c4e6e9fe210fafc7ca76c P 3f77db6ee4264d4c8ca1881c368b6f40 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c32998a2e494252852add6da4b30de3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43003 } } peers { permanent_uuid: "3f77db6ee4264d4c8ca1881c368b6f40" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42377 } } peers { permanent_uuid: "6951a644221047f9b7653878dd24b922" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39893 } }
I20260924 09:57:55.398579 11706 catalog_manager.cc:5495] T b74539682c2c4e6e9fe210fafc7ca76c P 3f77db6ee4264d4c8ca1881c368b6f40 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3f77db6ee4264d4c8ca1881c368b6f40 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "3f77db6ee4264d4c8ca1881c368b6f40" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c32998a2e494252852add6da4b30de3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43003 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3f77db6ee4264d4c8ca1881c368b6f40" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42377 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6951a644221047f9b7653878dd24b922" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39893 } health_report { overall_health: UNKNOWN } } }
I20260924 09:57:55.415808 12212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:57:55.411076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34058 (local address 127.0.97.1:42377)
0924 09:57:55.411287 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0924 09:57:55.411293 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 09:57:55.411310 (+ 17us) server_negotiation.cc:407] Connection header received
0924 09:57:55.411415 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:57:55.411418 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:57:55.411507 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:57:55.411626 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 09:57:55.411992 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:57:55.412686 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:57:55.413971 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:57:55.414339 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:57:55.414422 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:57:55.414814 (+ 392us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 09:57:55.415313 (+ 499us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 09:57:55.415411 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0924 09:57:55.415506 (+ 95us) server_negotiation.cc:299] Negotiation successful
0924 09:57:55.415569 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 09:57:55.423859 12018 raft_consensus.cc:1261] T b74539682c2c4e6e9fe210fafc7ca76c P 3c32998a2e494252852add6da4b30de3 [term 1 FOLLOWER]: Refusing update from remote peer 3f77db6ee4264d4c8ca1881c368b6f40: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 09:57:55.423859 12152 raft_consensus.cc:1261] T b74539682c2c4e6e9fe210fafc7ca76c P 6951a644221047f9b7653878dd24b922 [term 1 FOLLOWER]: Refusing update from remote peer 3f77db6ee4264d4c8ca1881c368b6f40: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 09:57:55.424871 12226 consensus_queue.cc:1035] T b74539682c2c4e6e9fe210fafc7ca76c P 3f77db6ee4264d4c8ca1881c368b6f40 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c32998a2e494252852add6da4b30de3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 09:57:55.425164 12222 consensus_queue.cc:1035] T b74539682c2c4e6e9fe210fafc7ca76c P 3f77db6ee4264d4c8ca1881c368b6f40 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6951a644221047f9b7653878dd24b922" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 09:57:55.435662 12064 tablet.cc:2369] T b74539682c2c4e6e9fe210fafc7ca76c P 3c32998a2e494252852add6da4b30de3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 09:57:55.450058 11706 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35246:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 09:57:55.450330 11706 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35246:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 09:57:55.453786 11706 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 42a9029a7aec461eb55f998a506b41f7: Sending DeleteTablet for 3 replicas of tablet b74539682c2c4e6e9fe210fafc7ca76c
I20260924 09:57:55.454972 11855 tablet_service.cc:1514] Processing DeleteTablet for tablet b74539682c2c4e6e9fe210fafc7ca76c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 09:57:55 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:34044
I20260924 09:57:55.455008 12132 tablet_service.cc:1514] Processing DeleteTablet for tablet b74539682c2c4e6e9fe210fafc7ca76c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 09:57:55 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:54186
I20260924 09:57:55.455081 11998 tablet_service.cc:1514] Processing DeleteTablet for tablet b74539682c2c4e6e9fe210fafc7ca76c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 09:57:55 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:44828
I20260924 09:57:55.457463 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu pid 11787
I20260924 09:57:55.467473 12241 tablet_replica.cc:330] T b74539682c2c4e6e9fe210fafc7ca76c P 3c32998a2e494252852add6da4b30de3: stopping tablet replica
I20260924 09:57:55.468082 12241 raft_consensus.cc:2229] T b74539682c2c4e6e9fe210fafc7ca76c P 3c32998a2e494252852add6da4b30de3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 09:57:55.468696 12241 raft_consensus.cc:2258] T b74539682c2c4e6e9fe210fafc7ca76c P 3c32998a2e494252852add6da4b30de3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 09:57:55.470924 12241 ts_tablet_manager.cc:1905] T b74539682c2c4e6e9fe210fafc7ca76c P 3c32998a2e494252852add6da4b30de3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 09:57:55.473179 12239 tablet_replica.cc:330] T b74539682c2c4e6e9fe210fafc7ca76c P 6951a644221047f9b7653878dd24b922: stopping tablet replica
I20260924 09:57:55.473918 12240 tablet_replica.cc:330] T b74539682c2c4e6e9fe210fafc7ca76c P 3f77db6ee4264d4c8ca1881c368b6f40: stopping tablet replica
I20260924 09:57:55.474424 12239 raft_consensus.cc:2229] T b74539682c2c4e6e9fe210fafc7ca76c P 6951a644221047f9b7653878dd24b922 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 09:57:55.475003 12240 raft_consensus.cc:2229] T b74539682c2c4e6e9fe210fafc7ca76c P 3f77db6ee4264d4c8ca1881c368b6f40 [term 1 LEADER]: Raft consensus shutting down.
I20260924 09:57:55.475628 12239 raft_consensus.cc:2258] T b74539682c2c4e6e9fe210fafc7ca76c P 6951a644221047f9b7653878dd24b922 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 09:57:55.476415 12240 raft_consensus.cc:2258] T b74539682c2c4e6e9fe210fafc7ca76c P 3f77db6ee4264d4c8ca1881c368b6f40 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 09:57:55.477968 12241 ts_tablet_manager.cc:1918] T b74539682c2c4e6e9fe210fafc7ca76c P 3c32998a2e494252852add6da4b30de3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 09:57:55.478171 12241 log.cc:1196] T b74539682c2c4e6e9fe210fafc7ca76c P 3c32998a2e494252852add6da4b30de3: Deleting WAL directory at /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790243735634835-388-0/minicluster-data/ts-1/wal/wals/b74539682c2c4e6e9fe210fafc7ca76c
I20260924 09:57:55.478801 12241 ts_tablet_manager.cc:1939] T b74539682c2c4e6e9fe210fafc7ca76c P 3c32998a2e494252852add6da4b30de3: Deleting consensus metadata
I20260924 09:57:55.480717 12240 ts_tablet_manager.cc:1905] T b74539682c2c4e6e9fe210fafc7ca76c P 3f77db6ee4264d4c8ca1881c368b6f40: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 09:57:55.480731 11694 catalog_manager.cc:4841] TS 3c32998a2e494252852add6da4b30de3 (127.0.97.2:43003): tablet b74539682c2c4e6e9fe210fafc7ca76c (table test-table [id=228585f5e5954a3f912c3df36c4a976c]) successfully deleted
I20260924 09:57:55.481056 12239 ts_tablet_manager.cc:1905] T b74539682c2c4e6e9fe210fafc7ca76c P 6951a644221047f9b7653878dd24b922: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 09:57:55.487138 12239 ts_tablet_manager.cc:1918] T b74539682c2c4e6e9fe210fafc7ca76c P 6951a644221047f9b7653878dd24b922: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
Sep 24 09:57:55 dist-test-slave-w8d2 krb5kdc[11231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243871, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260924 09:57:55.487468 12239 log.cc:1196] T b74539682c2c4e6e9fe210fafc7ca76c P 6951a644221047f9b7653878dd24b922: Deleting WAL directory at /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790243735634835-388-0/minicluster-data/ts-2/wal/wals/b74539682c2c4e6e9fe210fafc7ca76c
I20260924 09:57:55.488046 12239 ts_tablet_manager.cc:1939] T b74539682c2c4e6e9fe210fafc7ca76c P 6951a644221047f9b7653878dd24b922: Deleting consensus metadata
I20260924 09:57:55.489882 11692 catalog_manager.cc:4841] TS 6951a644221047f9b7653878dd24b922 (127.0.97.3:39893): tablet b74539682c2c4e6e9fe210fafc7ca76c (table test-table [id=228585f5e5954a3f912c3df36c4a976c]) successfully deleted
I20260924 09:57:55.490867 12240 ts_tablet_manager.cc:1918] T b74539682c2c4e6e9fe210fafc7ca76c P 3f77db6ee4264d4c8ca1881c368b6f40: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 09:57:55.491050 12240 log.cc:1196] T b74539682c2c4e6e9fe210fafc7ca76c P 3f77db6ee4264d4c8ca1881c368b6f40: Deleting WAL directory at /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790243735634835-388-0/minicluster-data/ts-0/wal/wals/b74539682c2c4e6e9fe210fafc7ca76c
I20260924 09:57:55.491627 12240 ts_tablet_manager.cc:1939] T b74539682c2c4e6e9fe210fafc7ca76c P 3f77db6ee4264d4c8ca1881c368b6f40: Deleting consensus metadata
I20260924 09:57:55.493475 11691 catalog_manager.cc:4841] TS 3f77db6ee4264d4c8ca1881c368b6f40 (127.0.97.1:42377): tablet b74539682c2c4e6e9fe210fafc7ca76c (table test-table [id=228585f5e5954a3f912c3df36c4a976c]) successfully deleted
I20260924 09:57:55.495069 12212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:57:55.459022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34072 (local address 127.0.97.1:42377)
0924 09:57:55.459194 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0924 09:57:55.459200 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 09:57:55.477357 (+ 18157us) server_negotiation.cc:407] Connection header received
0924 09:57:55.477698 (+ 341us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:57:55.477704 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:57:55.477789 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:57:55.477867 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:57:55.479783 (+ 1916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:57:55.480930 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:57:55.482800 (+ 1870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:57:55.483468 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:57:55.483612 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:57:55.488582 (+ 4970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:57:55.488601 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:57:55.488620 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:57:55.488669 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:57:55.492328 (+ 3659us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:57:55.493021 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:57:55.493028 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:57:55.493030 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:57:55.493185 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:57:55.493596 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:57:55.493601 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:57:55.493604 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:57:55.493953 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:57:55.494220 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0924 09:57:55.494542 (+ 322us) server_negotiation.cc:299] Negotiation successful
0924 09:57:55.494755 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260924 09:57:55.496277 11915 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 09:57:55.594204 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu with pid 11787
I20260924 09:57:55.609946 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu pid 11927
Sep 24 09:57:55 dist-test-slave-w8d2 krb5kdc[11231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243871, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260924 09:57:55.623683 12211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:57:55.610685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44834 (local address 127.0.97.2:43003)
0924 09:57:55.610836 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0924 09:57:55.610840 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 09:57:55.610858 (+ 18us) server_negotiation.cc:407] Connection header received
0924 09:57:55.610997 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:57:55.611000 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:57:55.611060 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:57:55.611137 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:57:55.612434 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:57:55.613128 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:57:55.614399 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:57:55.614763 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:57:55.614845 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:57:55.618631 (+ 3786us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:57:55.618645 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:57:55.618660 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:57:55.618696 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:57:55.621351 (+ 2655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:57:55.622013 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:57:55.622019 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:57:55.622021 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:57:55.622138 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:57:55.622422 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:57:55.622425 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:57:55.622426 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:57:55.622775 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:57:55.622983 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0924 09:57:55.623196 (+ 213us) server_negotiation.cc:299] Negotiation successful
0924 09:57:55.623386 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 09:57:55.624313 12058 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 09:57:55.710716 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu with pid 11927
I20260924 09:57:55.723794 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu pid 12069
Sep 24 09:57:55 dist-test-slave-w8d2 krb5kdc[11231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243871, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260924 09:57:55.737145 12214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:57:55.724555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54192 (local address 127.0.97.3:39893)
0924 09:57:55.724704 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0924 09:57:55.724708 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 09:57:55.724726 (+ 18us) server_negotiation.cc:407] Connection header received
0924 09:57:55.724779 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:57:55.724783 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:57:55.724842 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:57:55.724929 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:57:55.726102 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:57:55.726838 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:57:55.728141 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:57:55.728526 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:57:55.728615 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:57:55.732447 (+ 3832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:57:55.732469 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:57:55.732487 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:57:55.732520 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:57:55.734932 (+ 2412us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:57:55.735591 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:57:55.735598 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:57:55.735601 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:57:55.735707 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:57:55.735968 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:57:55.735970 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:57:55.735982 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:57:55.736314 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:57:55.736513 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0924 09:57:55.736748 (+ 235us) server_negotiation.cc:299] Negotiation successful
0924 09:57:55.736896 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 09:57:55.737706 12192 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 09:57:55.824832 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu with pid 12069
I20260924 09:57:55.837723 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu pid 11675
I20260924 09:57:55.838860 11737 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 09:57:55.946672 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu with pid 11675
I20260924 09:57:55.962741 388 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260924 09:57:56.296909 388 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260924 09:57:56.297118 388 mini_ranger.cc:66] Stopping Ranger...
I20260924 09:57:56.631265 388 mini_ranger.cc:68] Stopped Ranger
2026-09-24 09:57:56.631 UTC [11261] LOG: received smart shutdown request
2026-09-24 09:57:56.634 UTC [11261] LOG: background worker "logical replication launcher" (PID 11270) exited with exit code 1
2026-09-24 09:57:56.634 UTC [11265] LOG: shutting down
2026-09-24 09:57:56.764 UTC [11261] LOG: database system is shut down
2026-09-24T09:57:56Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (46213 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790243735634835-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 09:57:56 dist-test-slave-w8d2 krb5kdc[12255](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 09:57:56 dist-test-slave-w8d2 krb5kdc[12255](info): set up 2 sockets
Sep 24 09:57:56 dist-test-slave-w8d2 krb5kdc[12255](info): commencing operation
krb5kdc: starting...
W20260924 09:57:57.879279 388 mini_kdc.cc:121] Time spent starting KDC: real 0.974s user 0.000s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790243735634835-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790243735634835-388-0/krb5kdc/test-user.keytab.
Sep 24 09:57:58 dist-test-slave-w8d2 krb5kdc[12255](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243878, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T09:57:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T09:57:58Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790243735634835-388-0/krb5kdc/rangeradmin_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790243735634835-388-0/krb5kdc/rangeradmin_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790243735634835-388-0/krb5kdc/rangerlookup_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790243735634835-388-0/krb5kdc/rangerlookup_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790243735634835-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790243735634835-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
I20260924 09:57:58.166002 388 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790243735634835-388-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskPt_ZmK/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790243735634835-388-0/minicluster-data/postgres -l logfile start
2026-09-24 09:58:00.347 UTC [12285] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-24 09:58:00.347 UTC [12285] LOG: listening on IPv4 address "127.0.97.20", port 43299
2026-09-24 09:58:00.350 UTC [12285] LOG: listening on Unix socket "/tmp/.s.PGSQL.43299"
2026-09-24 09:58:00.370 UTC [12288] LOG: database system was shut down at 2026-09-24 09:57:59 UTC
2026-09-24 09:58:00.372 UTC [12285] LOG: database system is ready to accept connections
I20260924 09:58:01.259052 388 mini_postgres.cc:96] Postgres bound to 43299
2026-09-24 09:58:01.274 UTC [12296] FATAL: database "slave" does not exist
127.0.97.20:43299 - accepting connections
I20260924 09:58:01.274986 388 mini_ranger.cc:161] Starting Ranger...
I20260924 09:58:01.298050 388 mini_ranger.cc:84] Created miniranger Postgres user
I20260924 09:58:01.698554 388 mini_ranger.cc:87] Created ranger Postgres database
I20260924 09:58:01.698647 388 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790243735634835-388-0/minicluster-data/ranger-admin
2026-09-24 09:58:01,968 [I] DB FLAVOR :POSTGRES
2026-09-24 09:58:01,969 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 09:58:01,969 [I] Checking connection..
2026-09-24 09:58:01,969 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPt_ZmK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskPt_ZmK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:43299/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-24 09:58:02,211 [I] Checking connection passed.
2026-09-24 09:58:02,211 [I] --------- Verifying version history table ---------
2026-09-24 09:58:02,211 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPt_ZmK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskPt_ZmK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:43299/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 09:58:02,450 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 09:58:02,451 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPt_ZmK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskPt_ZmK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:43299/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 09:58:02,700 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 09:58:02,701 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-24 09:58:02,701 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPt_ZmK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskPt_ZmK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:43299/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskPt_ZmK/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-24 09:58:02.932 UTC [12360] WARNING: there is no transaction in progress
2026-09-24 09:58:02,938 [I] create_dbversion_catalog.sql file imported successfully
2026-09-24 09:58:02,938 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPt_ZmK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskPt_ZmK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:43299/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 09:58:03,181 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-24 09:58:03,182 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 09:58:03,182 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPt_ZmK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskPt_ZmK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:43299/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-24 09:58:03,418 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPt_ZmK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskPt_ZmK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:43299/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-24 09:58:03,656 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPt_ZmK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskPt_ZmK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:43299/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-w8d2.c.gcp-upstream.internal','N') ;"
2026-09-24 09:58:03,891 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPt_ZmK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskPt_ZmK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:43299/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 09:58:04,125 [I] Table x_portal_user does not exist in database ranger
2026-09-24 09:58:04,126 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPt_ZmK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskPt_ZmK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:43299/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 09:58:04,372 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-24 09:58:04,372 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPt_ZmK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskPt_ZmK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:43299/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 09:58:04,603 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPt_ZmK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskPt_ZmK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:43299/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 09:58:04,842 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-24 09:58:04,842 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPt_ZmK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskPt_ZmK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:43299/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskPt_ZmK/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-24 09:58:05.724 UTC [12468] WARNING: there is no transaction in progress
2026-09-24 09:58:05.739 UTC [12468] WARNING: there is no transaction in progress
2026-09-24 09:58:05.755 UTC [12468] WARNING: there is no transaction in progress
2026-09-24 09:58:05.769 UTC [12468] WARNING: there is no transaction in progress
2026-09-24 09:58:05.786 UTC [12468] WARNING: there is no transaction in progress
2026-09-24 09:58:05.803 UTC [12468] WARNING: there is no transaction in progress
2026-09-24 09:58:05.855 UTC [12468] WARNING: there is no transaction in progress
2026-09-24 09:58:05.861 UTC [12468] WARNING: there is no transaction in progress
2026-09-24 09:58:05.874 UTC [12468] WARNING: there is no transaction in progress
2026-09-24 09:58:05.883 UTC [12468] WARNING: there is no transaction in progress
2026-09-24 09:58:05.893 UTC [12468] WARNING: there is no transaction in progress
2026-09-24 09:58:05.905 UTC [12468] WARNING: there is no transaction in progress
2026-09-24 09:58:05.914 UTC [12468] WARNING: there is no transaction in progress
2026-09-24 09:58:05.919 UTC [12468] WARNING: there is no transaction in progress
2026-09-24 09:58:05.925 UTC [12468] WARNING: there is no transaction in progress
2026-09-24 09:58:05.930 UTC [12468] WARNING: there is no transaction in progress
2026-09-24 09:58:06,475 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-24 09:58:06,475 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPt_ZmK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskPt_ZmK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:43299/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-24 09:58:06,699 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-24 09:58:06,699 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPt_ZmK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskPt_ZmK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:43299/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 09:58:06,951 [I] Table x_portal_user already exists in database 'ranger'
2026-09-24 09:58:06,952 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPt_ZmK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskPt_ZmK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:43299/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 09:58:07,187 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-24 09:58:07,187 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPt_ZmK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskPt_ZmK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:43299/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 09:58:07,406 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPt_ZmK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskPt_ZmK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:43299/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 09:58:07,656 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPt_ZmK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskPt_ZmK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:43299/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-w8d2.c.gcp-upstream.internal';"
2026-09-24 09:58:07,897 [I] CORE_DB_SCHEMA import status has been updated
2026-09-24 09:58:07,898 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPt_ZmK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskPt_ZmK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:43299/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-24 09:58:08,148 [I] DB_PATCHES have already been applied
I20260924 09:58:08.153787 388 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790243735634835-388-0/minicluster-data/ranger-admin:/tmp/dist-test-taskPt_ZmK/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskPt_ZmK/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskPt_ZmK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskPt_ZmK/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260924 09:58:08.153869 388 mini_ranger.cc:193] Using host: 127.0.97.20
I20260924 09:58:08.166590 388 mini_ranger.cc:227] Ranger admin URL: http://127.0.97.20:40777
Sep 24, 2026 9:58:08 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 9:58:08 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 9:58:08 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790243735634835-388-0/minicluster-data/ranger-admin/ews/webapp
Sep 24, 2026 9:58:08 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790243735634835-388-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 24, 2026 9:58:08 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790243735634835-388-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 24, 2026 9:58:09 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790243735634835-388-0/minicluster-data/ranger-admin/ews/webapp].
Sep 24, 2026 9:58:09 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260924 09:58:09.177209 388 mini_ranger.cc:160] Time spent starting Ranger: real 7.902s user 0.002s sys 0.034s
Sep 24, 2026 9:58:28 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 24, 2026 9:58:28 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 24, 2026 9:58:28 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 24, 2026 9:58:28 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 24, 2026 9:58:28 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 24, 2026 9:58:28 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 24, 2026 9:58:28 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 24, 2026 9:58:28 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 24, 2026 9:58:28 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 24, 2026 9:58:28 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 24, 2026 9:58:28 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 24, 2026 9:58:28 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 24, 2026 9:58:28 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 24, 2026 9:58:28 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 24, 2026 9:58:28 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 24, 2026 9:58:28 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 24, 2026 9:58:28 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 24, 2026 9:58:29 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260924 09:58:30.027861 388 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790243735634835-388-0/krb5kdc/rangerkms_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790243735634835-388-0/krb5kdc/rangerkms_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.97.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.97.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790243735634835-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790243735634835-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 24 09:58:30 dist-test-slave-w8d2 krb5kdc[12255](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243910, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260924 09:58:30.191027 388 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260924 09:58:30.212033 388 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260924 09:58:30.965976 388 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260924 09:58:30.966086 388 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790243735634835-388-0/minicluster-data/ranger-kms
I20260924 09:58:30.966130 388 mini_ranger_kms.cc:227] Using postgres at 127.0.97.20:43299
2026-09-24 09:58:31,292 [I] DB FLAVOR :POSTGRES
2026-09-24 09:58:31,292 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 09:58:31,292 [I] Checking connection
2026-09-24 09:58:31,292 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPt_ZmK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790243735634835-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:43299/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-24 09:58:31,518 [I] connection success
2026-09-24 09:58:31,518 [I] --------- Verifying Ranger DB tables ---------
2026-09-24 09:58:31,518 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-24 09:58:31,518 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPt_ZmK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790243735634835-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:43299/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-24 09:58:31,746 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-24 09:58:31,746 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 09:58:31,747 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-24 09:58:31,747 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPt_ZmK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790243735634835-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:43299/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790243735634835-388-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-24 09:58:31,987 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260924 09:58:32.434082 388 mini_ranger_kms.cc:310] Created kms service
I20260924 09:58:32.638705 388 mini_ranger_kms.cc:326] Created kudu user
I20260924 09:58:32.681255 388 mini_ranger_kms.cc:343] Created rangerkms user
Sep 24 09:58:32 dist-test-slave-w8d2 krb5kdc[12255](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243910, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260924 09:58:33.073213 388 mini_ranger_kms.cc:384] Added ranger policy
I20260924 09:58:33.073551 388 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790243735634835-388-0/minicluster-data/ranger-kms:/tmp/dist-test-taskPt_ZmK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskPt_ZmK/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskPt_ZmK/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260924 09:58:33.073596 388 mini_ranger_kms.cc:240] Using host: 127.0.97.20
I20260924 09:58:33.089322 388 mini_ranger_kms.cc:276] Ranger KMS PID: 12648
I20260924 09:58:33.089449 388 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.97.20:44351
Sep 24, 2026 9:58:34 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 9:58:34 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 9:58:34 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790243735634835-388-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 24, 2026 9:58:34 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790243735634835-388-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 24, 2026 9:58:34 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790243735634835-388-0/minicluster-data/ranger-kms/ews/webapp].
Sep 24, 2026 9:58:34 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Sep 24 09:58:35 dist-test-slave-w8d2 krb5kdc[12255](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790243915, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 09:58:35.159358 388 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 4.968s user 0.101s sys 0.245s
I20260924 09:58:35.159610 388 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260924 09:58:35.159778 388 mini_ranger_kms.cc:401] 127.0.97.20:44351/kms/v1/keys
Sep 24 09:58:35 dist-test-slave-w8d2 krb5kdc[12255](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243915, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Sep 24 09:58:35 dist-test-slave-w8d2 krb5kdc[12255](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243915, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Sep 24 09:58:38 dist-test-slave-w8d2 krb5kdc[12255](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243918, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790243735634835-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790243735634835-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260924 09:58:38.159614 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
/tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790243735634835-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790243735634835-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790243735634835-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790243735634835-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42309
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790243735634835-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:42309
--ranger_config_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790243735634835-388-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:44351/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790243735634835-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790243735634835-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790243735634835-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 09:58:38.501067 12699 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 09:58:38.501367 12699 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 09:58:38.501493 12699 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 09:58:38.508509 12699 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 09:58:38.508607 12699 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 09:58:38.508677 12699 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 09:58:38.508752 12699 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 09:58:38.508819 12699 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 09:58:38.518942 12699 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790243735634835-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790243735634835-388-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.97.20:44351/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:42309
--ranger_config_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790243735634835-388-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790243735634835-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42309
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790243735634835-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790243735634835-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 09:58:38.520241 12699 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 09:58:38.521579 12699 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 09:58:38.532797 12705 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:58:38.532853 12707 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:58:38.532853 12704 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 09:58:38.533896 12699 server_base.cc:1034] running on GCE node
I20260924 09:58:38.534649 12699 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 09:58:38.536130 12699 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 09:58:38.537318 12699 hybrid_clock.cc:648] HybridClock initialized: now 1790243918537274 us; error 37 us; skew 500 ppm
I20260924 09:58:38.537644 12699 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 09:58:38 dist-test-slave-w8d2 krb5kdc[12255](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243918, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 09:58:38.540755 12699 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260924 09:58:38.542220 12699 webserver.cc:460] Webserver started at http://127.0.97.62:44679/ using document root <none> and password file <none>
I20260924 09:58:38.542845 12699 fs_manager.cc:359] Metadata directory not provided
I20260924 09:58:38.542945 12699 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 09:58:38.543215 12699 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 09:58:38 dist-test-slave-w8d2 krb5kdc[12255](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243918, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260924 09:58:38.602767 12699 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790243735634835-388-0/minicluster-data/master-0/data/instance:
uuid: "c7230627fb3d4044b420787a568b1443"
format_stamp: "Formatted at 2026-09-24 09:58:38 on dist-test-slave-w8d2"
server_key: "51a3c7e93ba9cfe85a887eebc3350549"
server_key_iv: "75dad42d5989f7f62458fe178f5ba292"
server_key_version: "kuduclusterkey@0"
I20260924 09:58:38.603614 12699 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790243735634835-388-0/minicluster-data/master-0/wal/instance:
uuid: "c7230627fb3d4044b420787a568b1443"
format_stamp: "Formatted at 2026-09-24 09:58:38 on dist-test-slave-w8d2"
server_key: "51a3c7e93ba9cfe85a887eebc3350549"
server_key_iv: "75dad42d5989f7f62458fe178f5ba292"
server_key_version: "kuduclusterkey@0"
I20260924 09:58:38.608914 12699 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260924 09:58:38.628285 12714 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 09:58:38.629855 12699 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260924 09:58:38.630038 12699 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790243735634835-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790243735634835-388-0/minicluster-data/master-0/wal
uuid: "c7230627fb3d4044b420787a568b1443"
format_stamp: "Formatted at 2026-09-24 09:58:38 on dist-test-slave-w8d2"
server_key: "51a3c7e93ba9cfe85a887eebc3350549"
server_key_iv: "75dad42d5989f7f62458fe178f5ba292"
server_key_version: "kuduclusterkey@0"
I20260924 09:58:38.630237 12699 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790243735634835-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790243735634835-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790243735634835-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 09:58:38.648124 12699 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 09:58:38.651798 12699 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 09:58:38.652061 12699 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 09:58:38.668009 12699 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:42309
I20260924 09:58:38.668037 12765 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:42309 every 8 connection(s)
I20260924 09:58:38.669246 12699 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790243735634835-388-0/minicluster-data/master-0/data/info.pb
I20260924 09:58:38.672690 12766 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 09:58:38.674356 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu as pid 12699
I20260924 09:58:38.674557 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790243735634835-388-0/minicluster-data/master-0/wal/instance
I20260924 09:58:38.675181 388 external_mini_cluster.cc:1428] Setting key 7b89edc31183e5c270a254c1e91f2f63
I20260924 09:58:38.683883 12766 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c7230627fb3d4044b420787a568b1443: Bootstrap starting.
I20260924 09:58:38.687434 12766 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c7230627fb3d4044b420787a568b1443: Neither blocks nor log segments found. Creating new log.
I20260924 09:58:38.688566 12766 log.cc:824] T 00000000000000000000000000000000 P c7230627fb3d4044b420787a568b1443: Log is configured to *not* fsync() on all Append() calls
Sep 24 09:58:38 dist-test-slave-w8d2 krb5kdc[12255](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243918, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 09:58:38.691151 12766 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c7230627fb3d4044b420787a568b1443: No bootstrap required, opened a new log
I20260924 09:58:38.697114 12769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:58:38.677108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35296 (local address 127.0.97.62:42309)
0924 09:58:38.677862 (+ 754us) server_negotiation.cc:206] Beginning negotiation
0924 09:58:38.677873 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 09:58:38.677913 (+ 40us) server_negotiation.cc:407] Connection header received
0924 09:58:38.679562 (+ 1649us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:58:38.679584 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:58:38.679961 (+ 377us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:58:38.680266 (+ 305us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:58:38.681804 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:58:38.683072 (+ 1268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:58:38.684726 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:58:38.685364 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:58:38.685490 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:58:38.689729 (+ 4239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:58:38.689737 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:58:38.689751 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:58:38.689797 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:58:38.693105 (+ 3308us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:58:38.693683 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:58:38.693692 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:58:38.693700 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:58:38.693899 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:58:38.694189 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:58:38.694194 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:58:38.694197 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:58:38.694841 (+ 644us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:58:38.695131 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0924 09:58:38.695431 (+ 300us) server_negotiation.cc:299] Negotiation successful
0924 09:58:38.695790 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":548,"thread_start_us":358,"threads_started":1}
I20260924 09:58:38.700645 12766 raft_consensus.cc:348] T 00000000000000000000000000000000 P c7230627fb3d4044b420787a568b1443 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7230627fb3d4044b420787a568b1443" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42309 } }
I20260924 09:58:38.700946 12766 raft_consensus.cc:374] T 00000000000000000000000000000000 P c7230627fb3d4044b420787a568b1443 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 09:58:38.701014 12766 raft_consensus.cc:729] T 00000000000000000000000000000000 P c7230627fb3d4044b420787a568b1443 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7230627fb3d4044b420787a568b1443, State: Initialized, Role: FOLLOWER
I20260924 09:58:38.701506 12766 consensus_queue.cc:260] T 00000000000000000000000000000000 P c7230627fb3d4044b420787a568b1443 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7230627fb3d4044b420787a568b1443" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42309 } }
I20260924 09:58:38.701704 12766 raft_consensus.cc:388] T 00000000000000000000000000000000 P c7230627fb3d4044b420787a568b1443 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 09:58:38.701792 12766 raft_consensus.cc:482] T 00000000000000000000000000000000 P c7230627fb3d4044b420787a568b1443 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 09:58:38.701903 12766 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c7230627fb3d4044b420787a568b1443 [term 0 FOLLOWER]: Advancing to term 1
I20260924 09:58:38.703896 12766 raft_consensus.cc:504] T 00000000000000000000000000000000 P c7230627fb3d4044b420787a568b1443 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7230627fb3d4044b420787a568b1443" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42309 } }
I20260924 09:58:38.704293 12766 leader_election.cc:302] T 00000000000000000000000000000000 P c7230627fb3d4044b420787a568b1443 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c7230627fb3d4044b420787a568b1443; no voters:
I20260924 09:58:38.704885 12766 leader_election.cc:288] T 00000000000000000000000000000000 P c7230627fb3d4044b420787a568b1443 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 09:58:38.705008 12771 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c7230627fb3d4044b420787a568b1443 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 09:58:38.705667 12771 raft_consensus.cc:686] T 00000000000000000000000000000000 P c7230627fb3d4044b420787a568b1443 [term 1 LEADER]: Becoming Leader. State: Replica: c7230627fb3d4044b420787a568b1443, State: Running, Role: LEADER
I20260924 09:58:38.706152 12771 consensus_queue.cc:237] T 00000000000000000000000000000000 P c7230627fb3d4044b420787a568b1443 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7230627fb3d4044b420787a568b1443" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42309 } }
I20260924 09:58:38.706789 12766 sys_catalog.cc:564] T 00000000000000000000000000000000 P c7230627fb3d4044b420787a568b1443 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 09:58:38.708472 12772 sys_catalog.cc:455] T 00000000000000000000000000000000 P c7230627fb3d4044b420787a568b1443 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c7230627fb3d4044b420787a568b1443" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7230627fb3d4044b420787a568b1443" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42309 } } }
I20260924 09:58:38.708745 12772 sys_catalog.cc:458] T 00000000000000000000000000000000 P c7230627fb3d4044b420787a568b1443 [sys.catalog]: This master's current role is: LEADER
I20260924 09:58:38.710296 12777 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 09:58:38.710171 12773 sys_catalog.cc:455] T 00000000000000000000000000000000 P c7230627fb3d4044b420787a568b1443 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c7230627fb3d4044b420787a568b1443. Latest consensus state: current_term: 1 leader_uuid: "c7230627fb3d4044b420787a568b1443" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7230627fb3d4044b420787a568b1443" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42309 } } }
I20260924 09:58:38.710788 12773 sys_catalog.cc:458] T 00000000000000000000000000000000 P c7230627fb3d4044b420787a568b1443 [sys.catalog]: This master's current role is: LEADER
I20260924 09:58:38.715545 12777 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 09:58:38.716082 12766 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790243735634835-388-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260924 09:58:38.731901 12777 catalog_manager.cc:1347] Generated new cluster ID: e50fa497e20d46fd80e285dab4dbfc82
I20260924 09:58:38.732031 12777 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 09:58:38.769042 12777 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 09:58:38.770387 12777 catalog_manager.cc:1504] Loading token signing keys...
I20260924 09:58:38.776759 12777 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c7230627fb3d4044b420787a568b1443: Generated new TSK 0
I20260924 09:58:38.777594 12777 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790243735634835-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790243735634835-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260924 09:58:39.579555 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
/tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790243735634835-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790243735634835-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790243735634835-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790243735634835-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:44351/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42309
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790243735634835-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790243735634835-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790243735634835-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790243735634835-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 09:58:39 dist-test-slave-w8d2 krb5kdc[12255](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790243919, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260924 09:58:40.022670 12811 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 09:58:40.023061 12811 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 09:58:40.023270 12811 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 09:58:40.034814 12811 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 09:58:40.034969 12811 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 09:58:40.035202 12811 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260924 09:58:40.050884 12811 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790243735634835-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790243735634835-388-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.97.20:44351/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790243735634835-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790243735634835-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42309
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790243735634835-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 09:58:40.052886 12811 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 09:58:40.054730 12811 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 09:58:40.078749 12821 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:58:40.079648 12822 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:58:40.090551 12824 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 09:58:40.091070 12811 server_base.cc:1034] running on GCE node
I20260924 09:58:40.092103 12811 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 09:58:40.093650 12811 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 09:58:40.094954 12811 hybrid_clock.cc:648] HybridClock initialized: now 1790243920094855 us; error 95 us; skew 500 ppm
I20260924 09:58:40.095325 12811 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 09:58:40 dist-test-slave-w8d2 krb5kdc[12255](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243920, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 09:58:40.098543 12811 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260924 09:58:40.100338 12811 webserver.cc:460] Webserver started at http://127.0.97.1:44873/ using document root <none> and password file <none>
I20260924 09:58:40.101056 12811 fs_manager.cc:359] Metadata directory not provided
I20260924 09:58:40.101176 12811 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 09:58:40.101506 12811 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 09:58:40 dist-test-slave-w8d2 krb5kdc[12255](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243920, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260924 09:58:40.123600 12811 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790243735634835-388-0/minicluster-data/ts-0/data/instance:
uuid: "6675bb8ea82e4825b85aa9a50d9533d2"
format_stamp: "Formatted at 2026-09-24 09:58:40 on dist-test-slave-w8d2"
server_key: "e3632be09f1faec85388161d61582097"
server_key_iv: "ff04a505ab45fa8806c8519278b3a1f9"
server_key_version: "kuduclusterkey@0"
I20260924 09:58:40.124962 12811 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790243735634835-388-0/minicluster-data/ts-0/wal/instance:
uuid: "6675bb8ea82e4825b85aa9a50d9533d2"
format_stamp: "Formatted at 2026-09-24 09:58:40 on dist-test-slave-w8d2"
server_key: "e3632be09f1faec85388161d61582097"
server_key_iv: "ff04a505ab45fa8806c8519278b3a1f9"
server_key_version: "kuduclusterkey@0"
I20260924 09:58:40.158340 12811 fs_manager.cc:691] Time spent creating directory manager: real 0.033s user 0.007s sys 0.020s
I20260924 09:58:40.195884 12831 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 09:58:40.197940 12811 fs_manager.cc:725] Time spent opening block manager: real 0.019s user 0.005s sys 0.010s
I20260924 09:58:40.198261 12811 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790243735634835-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790243735634835-388-0/minicluster-data/ts-0/wal
uuid: "6675bb8ea82e4825b85aa9a50d9533d2"
format_stamp: "Formatted at 2026-09-24 09:58:40 on dist-test-slave-w8d2"
server_key: "e3632be09f1faec85388161d61582097"
server_key_iv: "ff04a505ab45fa8806c8519278b3a1f9"
server_key_version: "kuduclusterkey@0"
I20260924 09:58:40.198633 12811 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790243735634835-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790243735634835-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790243735634835-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 09:58:40.238301 12811 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 09:58:40.243260 12811 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 09:58:40.243667 12811 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 09:58:40.245532 12811 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 09:58:40.247104 12811 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 09:58:40.247184 12811 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 09:58:40.247346 12811 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 09:58:40.247400 12811 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 09:58:40 dist-test-slave-w8d2 krb5kdc[12255](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243919, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260924 09:58:40.312052 12811 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:40527
I20260924 09:58:40.312107 12943 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:40527 every 8 connection(s)
I20260924 09:58:40.313550 12811 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790243735634835-388-0/minicluster-data/ts-0/data/info.pb
I20260924 09:58:40.318696 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu as pid 12811
I20260924 09:58:40.318924 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790243735634835-388-0/minicluster-data/ts-0/wal/instance
I20260924 09:58:40.319780 388 external_mini_cluster.cc:1428] Setting key c94901cab53584e279a23c374b720abd
Sep 24 09:58:40 dist-test-slave-w8d2 krb5kdc[12255](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243920, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260924 09:58:40.357720 12947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:58:40.318079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:44059 (local address 127.0.97.62:42309)
0924 09:58:40.327556 (+ 9477us) server_negotiation.cc:206] Beginning negotiation
0924 09:58:40.327562 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 09:58:40.327584 (+ 22us) server_negotiation.cc:407] Connection header received
0924 09:58:40.327647 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:58:40.327652 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:58:40.327744 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:58:40.327855 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:58:40.336415 (+ 8560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:58:40.337507 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:58:40.340017 (+ 2510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:58:40.340655 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:58:40.340756 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:58:40.350099 (+ 9343us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:58:40.350107 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:58:40.350112 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:58:40.350159 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:58:40.353638 (+ 3479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:58:40.354723 (+ 1085us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:58:40.354730 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:58:40.354734 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:58:40.354896 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:58:40.355534 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:58:40.355540 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:58:40.355543 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:58:40.355921 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:58:40.356059 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0924 09:58:40.357111 (+ 1052us) server_negotiation.cc:299] Negotiation successful
0924 09:58:40.357350 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9348,"thread_start_us":9242,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260924 09:58:40.360009 12946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:58:40.317470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42309 (local address 127.0.97.1:44059)
0924 09:58:40.323995 (+ 6525us) negotiation.cc:107] Waiting for socket to connect
0924 09:58:40.324060 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0924 09:58:40.326714 (+ 2654us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:58:40.331658 (+ 4944us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:58:40.331672 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:58:40.332373 (+ 701us) client_negotiation.cc:189] Negotiated authn=SASL
0924 09:58:40.336157 (+ 3784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:58:40.336192 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:58:40.337735 (+ 1543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:58:40.337741 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:58:40.339818 (+ 2077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:58:40.339835 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:58:40.341108 (+ 1273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:58:40.341114 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:58:40.341414 (+ 300us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:58:40.346788 (+ 5374us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 09:58:40.346835 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0924 09:58:40.349697 (+ 2862us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 09:58:40.353860 (+ 4163us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 09:58:40.353871 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 09:58:40.353892 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0924 09:58:40.354563 (+ 671us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 09:58:40.355076 (+ 513us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 09:58:40.355080 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 09:58:40.355084 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 09:58:40.355397 (+ 313us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 09:58:40.356152 (+ 755us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 09:58:40.356161 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 09:58:40.356651 (+ 490us) client_negotiation.cc:769] Sending connection context
0924 09:58:40.357986 (+ 1335us) client_negotiation.cc:240] Negotiation successful
0924 09:58:40.358548 (+ 562us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6107,"thread_start_us":13202,"threads_started":1}
I20260924 09:58:40.363677 12944 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42309
I20260924 09:58:40.364050 12944 heartbeater.cc:464] Registering TS with master...
I20260924 09:58:40.364941 12944 heartbeater.cc:511] Master 127.0.97.62:42309 requested a full tablet report, sending...
I20260924 09:58:40.367687 12731 ts_manager.cc:194] Registered new tserver with Master: 6675bb8ea82e4825b85aa9a50d9533d2 (127.0.97.1:40527)
I20260924 09:58:40.369786 12731 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:44059
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790243735634835-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790243735634835-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260924 09:58:40.398922 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
/tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790243735634835-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790243735634835-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790243735634835-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790243735634835-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:44351/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42309
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790243735634835-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790243735634835-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790243735634835-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790243735634835-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 09:58:40 dist-test-slave-w8d2 krb5kdc[12255](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243919, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
W20260924 09:58:40.873816 12950 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 09:58:40.874305 12950 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 09:58:40.874481 12950 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 09:58:40.885730 12950 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 09:58:40.885957 12950 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 09:58:40.886173 12950 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260924 09:58:40.901563 12950 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790243735634835-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790243735634835-388-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.97.20:44351/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790243735634835-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790243735634835-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42309
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790243735634835-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 09:58:40.903839 12950 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 09:58:40.905989 12950 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 09:58:40.944849 12957 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:58:40.947665 12961 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:58:40.947692 12958 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 09:58:40.948808 12950 server_base.cc:1034] running on GCE node
I20260924 09:58:40.949746 12950 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 09:58:40.963614 12950 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 09:58:40.965004 12950 hybrid_clock.cc:648] HybridClock initialized: now 1790243920964895 us; error 114 us; skew 500 ppm
I20260924 09:58:40.965677 12950 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 09:58:40 dist-test-slave-w8d2 krb5kdc[12255](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243920, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 09:58:40.970887 12950 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260924 09:58:41.007781 12950 webserver.cc:460] Webserver started at http://127.0.97.2:34931/ using document root <none> and password file <none>
I20260924 09:58:41.008987 12950 fs_manager.cc:359] Metadata directory not provided
I20260924 09:58:41.009205 12950 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 09:58:41.009656 12950 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 09:58:41 dist-test-slave-w8d2 krb5kdc[12255](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243920, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260924 09:58:41.032845 12950 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790243735634835-388-0/minicluster-data/ts-1/data/instance:
uuid: "5053831bb8594125aebee1c74b106a25"
format_stamp: "Formatted at 2026-09-24 09:58:41 on dist-test-slave-w8d2"
server_key: "21f32927fe9187890712941dd6781983"
server_key_iv: "f27ef57b6790efd46e68f3a8ea316c18"
server_key_version: "kuduclusterkey@0"
I20260924 09:58:41.034086 12950 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790243735634835-388-0/minicluster-data/ts-1/wal/instance:
uuid: "5053831bb8594125aebee1c74b106a25"
format_stamp: "Formatted at 2026-09-24 09:58:41 on dist-test-slave-w8d2"
server_key: "21f32927fe9187890712941dd6781983"
server_key_iv: "f27ef57b6790efd46e68f3a8ea316c18"
server_key_version: "kuduclusterkey@0"
I20260924 09:58:41.057269 12950 fs_manager.cc:691] Time spent creating directory manager: real 0.023s user 0.013s sys 0.004s
I20260924 09:58:41.079442 12974 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 09:58:41.081787 12950 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.001s sys 0.000s
I20260924 09:58:41.082072 12950 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790243735634835-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790243735634835-388-0/minicluster-data/ts-1/wal
uuid: "5053831bb8594125aebee1c74b106a25"
format_stamp: "Formatted at 2026-09-24 09:58:41 on dist-test-slave-w8d2"
server_key: "21f32927fe9187890712941dd6781983"
server_key_iv: "f27ef57b6790efd46e68f3a8ea316c18"
server_key_version: "kuduclusterkey@0"
I20260924 09:58:41.082413 12950 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790243735634835-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790243735634835-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790243735634835-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 09:58:41.156128 12950 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 09:58:41.161918 12950 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 09:58:41.162278 12950 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 09:58:41.175855 12950 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 09:58:41.184476 12950 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 09:58:41.184676 12950 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260924 09:58:41.184857 12950 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 09:58:41.184949 12950 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 09:58:41.251070 12950 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:42709
I20260924 09:58:41.251114 13086 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:42709 every 8 connection(s)
I20260924 09:58:41.252848 12950 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790243735634835-388-0/minicluster-data/ts-1/data/info.pb
I20260924 09:58:41.253796 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu as pid 12950
I20260924 09:58:41.253971 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790243735634835-388-0/minicluster-data/ts-1/wal/instance
I20260924 09:58:41.254594 388 external_mini_cluster.cc:1428] Setting key 0bd9030dd4bbada32d38be37fc5233a9
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Sep 24 09:58:41 dist-test-slave-w8d2 krb5kdc[12255](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243920, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 09:58:41.298646 13091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:58:41.268808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42309 (local address 127.0.97.2:47967)
0924 09:58:41.271039 (+ 2231us) negotiation.cc:107] Waiting for socket to connect
0924 09:58:41.271101 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0924 09:58:41.273644 (+ 2543us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:58:41.274683 (+ 1039us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:58:41.274696 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:58:41.275360 (+ 664us) client_negotiation.cc:189] Negotiated authn=SASL
0924 09:58:41.276211 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:58:41.276236 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:58:41.280850 (+ 4614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:58:41.280856 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:58:41.283043 (+ 2187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:58:41.283062 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:58:41.284576 (+ 1514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:58:41.284587 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:58:41.284910 (+ 323us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:58:41.285742 (+ 832us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 09:58:41.285774 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0924 09:58:41.289624 (+ 3850us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 09:58:41.293647 (+ 4023us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 09:58:41.293658 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 09:58:41.293679 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0924 09:58:41.294309 (+ 630us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 09:58:41.294773 (+ 464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 09:58:41.294778 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 09:58:41.294782 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 09:58:41.295046 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 09:58:41.295861 (+ 815us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 09:58:41.295870 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 09:58:41.296331 (+ 461us) client_negotiation.cc:769] Sending connection context
0924 09:58:41.296635 (+ 304us) client_negotiation.cc:240] Negotiation successful
0924 09:58:41.297208 (+ 573us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1928,"thread_start_us":1678,"threads_started":1}
I20260924 09:58:41.299856 13090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:58:41.268030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:47967 (local address 127.0.97.62:42309)
0924 09:58:41.270652 (+ 2622us) server_negotiation.cc:206] Beginning negotiation
0924 09:58:41.270659 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 09:58:41.271207 (+ 548us) server_negotiation.cc:407] Connection header received
0924 09:58:41.274159 (+ 2952us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:58:41.274165 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:58:41.274254 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:58:41.274374 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:58:41.279405 (+ 5031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:58:41.280583 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:58:41.283291 (+ 2708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:58:41.284076 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:58:41.284232 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:58:41.289939 (+ 5707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:58:41.289947 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:58:41.289953 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:58:41.290007 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:58:41.293452 (+ 3445us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:58:41.294470 (+ 1018us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:58:41.294476 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:58:41.294480 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:58:41.294641 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:58:41.295187 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:58:41.295193 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:58:41.295196 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:58:41.295587 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:58:41.295777 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0924 09:58:41.299206 (+ 3429us) server_negotiation.cc:299] Negotiation successful
0924 09:58:41.299501 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2462,"thread_start_us":2325,"threads_started":1}
I20260924 09:58:41.301107 13087 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42309
I20260924 09:58:41.301394 13087 heartbeater.cc:464] Registering TS with master...
I20260924 09:58:41.302261 13087 heartbeater.cc:511] Master 127.0.97.62:42309 requested a full tablet report, sending...
I20260924 09:58:41.303996 12731 ts_manager.cc:194] Registered new tserver with Master: 5053831bb8594125aebee1c74b106a25 (127.0.97.2:42709)
I20260924 09:58:41.305151 12731 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:47967
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790243735634835-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790243735634835-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260924 09:58:41.322937 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
/tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790243735634835-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790243735634835-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790243735634835-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790243735634835-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:44351/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42309
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790243735634835-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790243735634835-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790243735634835-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790243735634835-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 09:58:41.373951 12944 heartbeater.cc:503] Master 127.0.97.62:42309 was elected leader, sending a full tablet report...
W20260924 09:58:41.684949 13093 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 09:58:41.685261 13093 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 09:58:41.685391 13093 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 09:58:41.692790 13093 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 09:58:41.692924 13093 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 09:58:41.693090 13093 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260924 09:58:41.703073 13093 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790243735634835-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790243735634835-388-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.97.20:44351/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790243735634835-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790243735634835-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42309
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790243735634835-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 09:58:41.704452 13093 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 09:58:41.705765 13093 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 09:58:41.718391 13098 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:58:41.718433 13101 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:58:41.718473 13099 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 09:58:41.719197 13093 server_base.cc:1034] running on GCE node
I20260924 09:58:41.719797 13093 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 09:58:41.720696 13093 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 09:58:41.721892 13093 hybrid_clock.cc:648] HybridClock initialized: now 1790243921721828 us; error 52 us; skew 500 ppm
I20260924 09:58:41.722188 13093 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 09:58:41 dist-test-slave-w8d2 krb5kdc[12255](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243921, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 09:58:41.725473 13093 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260924 09:58:41.726881 13093 webserver.cc:460] Webserver started at http://127.0.97.3:41599/ using document root <none> and password file <none>
I20260924 09:58:41.727576 13093 fs_manager.cc:359] Metadata directory not provided
I20260924 09:58:41.727701 13093 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 09:58:41.727982 13093 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 09:58:41 dist-test-slave-w8d2 krb5kdc[12255](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243921, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260924 09:58:41.742255 13093 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790243735634835-388-0/minicluster-data/ts-2/data/instance:
uuid: "ec84e06401aa46209a01c4d6fc74ba79"
format_stamp: "Formatted at 2026-09-24 09:58:41 on dist-test-slave-w8d2"
server_key: "b38eeb2472652491b15eb90fc8adcc38"
server_key_iv: "905018f113d09bca4a4b41385f9c1af1"
server_key_version: "kuduclusterkey@0"
I20260924 09:58:41.743126 13093 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790243735634835-388-0/minicluster-data/ts-2/wal/instance:
uuid: "ec84e06401aa46209a01c4d6fc74ba79"
format_stamp: "Formatted at 2026-09-24 09:58:41 on dist-test-slave-w8d2"
server_key: "b38eeb2472652491b15eb90fc8adcc38"
server_key_iv: "905018f113d09bca4a4b41385f9c1af1"
server_key_version: "kuduclusterkey@0"
I20260924 09:58:41.748564 13093 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260924 09:58:41.762914 13108 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 09:58:41.764490 13093 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260924 09:58:41.764693 13093 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790243735634835-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790243735634835-388-0/minicluster-data/ts-2/wal
uuid: "ec84e06401aa46209a01c4d6fc74ba79"
format_stamp: "Formatted at 2026-09-24 09:58:41 on dist-test-slave-w8d2"
server_key: "b38eeb2472652491b15eb90fc8adcc38"
server_key_iv: "905018f113d09bca4a4b41385f9c1af1"
server_key_version: "kuduclusterkey@0"
I20260924 09:58:41.764914 13093 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790243735634835-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790243735634835-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790243735634835-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 09:58:41.778141 13093 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 09:58:41.781682 13093 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 09:58:41.781953 13093 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 09:58:41.782868 13093 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 09:58:41.784204 13093 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 09:58:41.784272 13093 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 09:58:41.784379 13093 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 09:58:41.784425 13093 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 09:58:41.814390 13093 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:44923
I20260924 09:58:41.814424 13220 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:44923 every 8 connection(s)
I20260924 09:58:41.815758 13093 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790243735634835-388-0/minicluster-data/ts-2/data/info.pb
I20260924 09:58:41.816992 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu as pid 13093
I20260924 09:58:41.817138 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790243735634835-388-0/minicluster-data/ts-2/wal/instance
I20260924 09:58:41.817579 388 external_mini_cluster.cc:1428] Setting key 99a4c10e584f0ebb9b749325e287e612
Sep 24 09:58:41 dist-test-slave-w8d2 krb5kdc[12255](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243921, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 09:58:41.835115 13223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:58:41.818147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:42113 (local address 127.0.97.62:42309)
0924 09:58:41.818755 (+ 608us) server_negotiation.cc:206] Beginning negotiation
0924 09:58:41.818761 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 09:58:41.819739 (+ 978us) server_negotiation.cc:407] Connection header received
0924 09:58:41.821564 (+ 1825us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:58:41.821569 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:58:41.821647 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:58:41.821742 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:58:41.823068 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:58:41.823843 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:58:41.825365 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:58:41.825815 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:58:41.825922 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:58:41.829199 (+ 3277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:58:41.829204 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:58:41.829207 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:58:41.829241 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:58:41.831991 (+ 2750us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:58:41.832706 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:58:41.832711 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:58:41.832715 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:58:41.832851 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:58:41.833300 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:58:41.833305 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:58:41.833309 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:58:41.833581 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:58:41.833709 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0924 09:58:41.834529 (+ 820us) server_negotiation.cc:299] Negotiation successful
0924 09:58:41.834759 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":483,"thread_start_us":324,"threads_started":1}
I20260924 09:58:41.836112 13224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:58:41.818771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42309 (local address 127.0.97.3:42113)
0924 09:58:41.819591 (+ 820us) negotiation.cc:107] Waiting for socket to connect
0924 09:58:41.819644 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0924 09:58:41.821315 (+ 1671us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:58:41.821966 (+ 651us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:58:41.821973 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:58:41.822367 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0924 09:58:41.822917 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:58:41.822933 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:58:41.823979 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:58:41.823985 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:58:41.825217 (+ 1232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:58:41.825246 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:58:41.826026 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:58:41.826032 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:58:41.826280 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:58:41.826928 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 09:58:41.826951 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 09:58:41.828964 (+ 2013us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 09:58:41.832135 (+ 3171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 09:58:41.832145 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 09:58:41.832164 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0924 09:58:41.832596 (+ 432us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 09:58:41.832955 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 09:58:41.832960 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 09:58:41.832964 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 09:58:41.833202 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 09:58:41.833696 (+ 494us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 09:58:41.833704 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 09:58:41.834132 (+ 428us) client_negotiation.cc:769] Sending connection context
0924 09:58:41.834433 (+ 301us) client_negotiation.cc:240] Negotiation successful
0924 09:58:41.834963 (+ 530us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":560,"thread_start_us":344,"threads_started":1}
I20260924 09:58:41.837426 13221 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42309
I20260924 09:58:41.837705 13221 heartbeater.cc:464] Registering TS with master...
I20260924 09:58:41.838397 13221 heartbeater.cc:511] Master 127.0.97.62:42309 requested a full tablet report, sending...
I20260924 09:58:41.839757 12731 ts_manager.cc:194] Registered new tserver with Master: ec84e06401aa46209a01c4d6fc74ba79 (127.0.97.3:44923)
I20260924 09:58:41.840528 12731 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:42113
I20260924 09:58:41.844926 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 09:58:41.846570 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu pid 12811
Sep 24 09:58:41 dist-test-slave-w8d2 krb5kdc[12255](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243918, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260924 09:58:41.862752 13226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:58:41.847541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53076 (local address 127.0.97.1:40527)
0924 09:58:41.848153 (+ 612us) server_negotiation.cc:206] Beginning negotiation
0924 09:58:41.848160 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 09:58:41.848184 (+ 24us) server_negotiation.cc:407] Connection header received
0924 09:58:41.848249 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:58:41.848255 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:58:41.848462 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:58:41.848590 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:58:41.849979 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:58:41.851036 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:58:41.852466 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:58:41.853057 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:58:41.853165 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:58:41.857121 (+ 3956us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:58:41.857135 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:58:41.857150 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:58:41.857188 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:58:41.860266 (+ 3078us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:58:41.861009 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:58:41.861015 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:58:41.861018 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:58:41.861186 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:58:41.861487 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:58:41.861490 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:58:41.861492 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:58:41.861826 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:58:41.862047 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0924 09:58:41.862287 (+ 240us) server_negotiation.cc:299] Negotiation successful
0924 09:58:41.862484 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":462,"thread_start_us":305,"threads_started":1}
I20260924 09:58:41.864048 12939 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 09:58:41.944900 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu with pid 12811
I20260924 09:58:41.957123 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu pid 12950
Sep 24 09:58:41 dist-test-slave-w8d2 krb5kdc[12255](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243918, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260924 09:58:41.972797 13228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:58:41.957988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55588 (local address 127.0.97.2:42709)
0924 09:58:41.958520 (+ 532us) server_negotiation.cc:206] Beginning negotiation
0924 09:58:41.958527 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 09:58:41.958553 (+ 26us) server_negotiation.cc:407] Connection header received
0924 09:58:41.958613 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:58:41.958620 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:58:41.958855 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:58:41.958989 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:58:41.960220 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:58:41.961290 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:58:41.962532 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:58:41.963153 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:58:41.963370 (+ 217us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:58:41.967288 (+ 3918us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:58:41.967307 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:58:41.967325 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:58:41.967364 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:58:41.970265 (+ 2901us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:58:41.970944 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:58:41.970951 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:58:41.970954 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:58:41.971088 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:58:41.971442 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:58:41.971446 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:58:41.971448 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:58:41.971796 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:58:41.972020 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0924 09:58:41.972247 (+ 227us) server_negotiation.cc:299] Negotiation successful
0924 09:58:41.972445 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":388,"thread_start_us":278,"threads_started":1}
I20260924 09:58:41.973999 13082 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 09:58:42.055800 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu with pid 12950
I20260924 09:58:42.068745 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu pid 13093
Sep 24 09:58:42 dist-test-slave-w8d2 krb5kdc[12255](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243918, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260924 09:58:42.084836 13230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:58:42.069742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45862 (local address 127.0.97.3:44923)
0924 09:58:42.070366 (+ 624us) server_negotiation.cc:206] Beginning negotiation
0924 09:58:42.070373 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 09:58:42.070401 (+ 28us) server_negotiation.cc:407] Connection header received
0924 09:58:42.070457 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:58:42.070464 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:58:42.070676 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:58:42.070813 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:58:42.072074 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:58:42.073065 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:58:42.074217 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:58:42.074766 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:58:42.074857 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:58:42.079033 (+ 4176us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:58:42.079046 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:58:42.079062 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:58:42.079103 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:58:42.081907 (+ 2804us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:58:42.082644 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:58:42.082658 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:58:42.082661 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:58:42.082797 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:58:42.083182 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:58:42.083187 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:58:42.083190 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:58:42.083583 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:58:42.083886 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0924 09:58:42.084217 (+ 331us) server_negotiation.cc:299] Negotiation successful
0924 09:58:42.084477 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":485,"thread_start_us":343,"threads_started":1}
I20260924 09:58:42.086099 13216 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 09:58:42.167824 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu with pid 13093
I20260924 09:58:42.180437 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu pid 12699
I20260924 09:58:42.181741 12761 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 09:58:42.285445 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu with pid 12699
I20260924 09:58:42.302599 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
/tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790243735634835-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790243735634835-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790243735634835-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790243735634835-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42309
--webserver_interface=127.0.97.62
--webserver_port=44679
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790243735634835-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:42309
--ranger_config_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790243735634835-388-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:44351/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:44351/kms with env {KRB5CCNAME=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790243735634835-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790243735634835-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790243735634835-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 09:58:42.612195 13233 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 09:58:42.612459 13233 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 09:58:42.612579 13233 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 09:58:42.619691 13233 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 09:58:42.619798 13233 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 09:58:42.619867 13233 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 09:58:42.619933 13233 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 09:58:42.619997 13233 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 09:58:42.629940 13233 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790243735634835-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790243735634835-388-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:44351/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:42309
--ranger_config_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790243735634835-388-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790243735634835-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42309
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790243735634835-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=44679
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790243735634835-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 09:58:42.631191 13233 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 09:58:42.632486 13233 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 09:58:42.644761 13241 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:58:42.644757 13239 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:58:42.645002 13238 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 09:58:42.645416 13233 server_base.cc:1034] running on GCE node
I20260924 09:58:42.646206 13233 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 09:58:42.647645 13233 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 09:58:42.648838 13233 hybrid_clock.cc:648] HybridClock initialized: now 1790243922648793 us; error 39 us; skew 500 ppm
I20260924 09:58:42.649178 13233 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 09:58:42 dist-test-slave-w8d2 krb5kdc[12255](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243922, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 09:58:42.652055 13233 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260924 09:58:42.653429 13233 webserver.cc:460] Webserver started at http://127.0.97.62:44679/ using document root <none> and password file <none>
I20260924 09:58:42.654053 13233 fs_manager.cc:359] Metadata directory not provided
I20260924 09:58:42.654181 13233 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 09:58:42 dist-test-slave-w8d2 krb5kdc[12255](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243922, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260924 09:58:42.717335 13233 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20260924 09:58:42.720202 13249 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 09:58:42.721505 13233 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260924 09:58:42.721688 13233 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790243735634835-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790243735634835-388-0/minicluster-data/master-0/wal
uuid: "c7230627fb3d4044b420787a568b1443"
format_stamp: "Formatted at 2026-09-24 09:58:38 on dist-test-slave-w8d2"
server_key: "51a3c7e93ba9cfe85a887eebc3350549"
server_key_iv: "75dad42d5989f7f62458fe178f5ba292"
server_key_version: "kuduclusterkey@0"
I20260924 09:58:42.722596 13233 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790243735634835-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790243735634835-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790243735634835-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 09:58:42.744947 13233 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 09:58:42.748988 13233 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 09:58:42.749217 13233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 09:58:42.767889 13233 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:42309
I20260924 09:58:42.767910 13300 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:42309 every 8 connection(s)
I20260924 09:58:42.769238 13233 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790243735634835-388-0/minicluster-data/master-0/data/info.pb
I20260924 09:58:42.772414 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu as pid 13233
I20260924 09:58:42.773157 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
/tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790243735634835-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790243735634835-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790243735634835-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790243735634835-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:44351/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:40527
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:42309
--webserver_port=44873
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790243735634835-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:44351/kms with env {KRB5CCNAME=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790243735634835-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790243735634835-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790243735634835-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 09:58:42.775870 13301 sys_catalog.cc:263] Verifying existing consensus state
I20260924 09:58:42.779196 13301 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c7230627fb3d4044b420787a568b1443: Bootstrap starting.
I20260924 09:58:42.804596 13301 log.cc:824] T 00000000000000000000000000000000 P c7230627fb3d4044b420787a568b1443: Log is configured to *not* fsync() on all Append() calls
I20260924 09:58:42.815105 13301 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c7230627fb3d4044b420787a568b1443: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 09:58:42.815722 13301 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c7230627fb3d4044b420787a568b1443: Bootstrap complete.
I20260924 09:58:42.828336 13301 raft_consensus.cc:348] T 00000000000000000000000000000000 P c7230627fb3d4044b420787a568b1443 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7230627fb3d4044b420787a568b1443" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42309 } }
I20260924 09:58:42.829515 13301 raft_consensus.cc:729] T 00000000000000000000000000000000 P c7230627fb3d4044b420787a568b1443 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7230627fb3d4044b420787a568b1443, State: Initialized, Role: FOLLOWER
I20260924 09:58:42.830209 13301 consensus_queue.cc:260] T 00000000000000000000000000000000 P c7230627fb3d4044b420787a568b1443 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7230627fb3d4044b420787a568b1443" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42309 } }
I20260924 09:58:42.830458 13301 raft_consensus.cc:388] T 00000000000000000000000000000000 P c7230627fb3d4044b420787a568b1443 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 09:58:42.830578 13301 raft_consensus.cc:482] T 00000000000000000000000000000000 P c7230627fb3d4044b420787a568b1443 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 09:58:42.830794 13301 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c7230627fb3d4044b420787a568b1443 [term 1 FOLLOWER]: Advancing to term 2
I20260924 09:58:42.836525 13301 raft_consensus.cc:504] T 00000000000000000000000000000000 P c7230627fb3d4044b420787a568b1443 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7230627fb3d4044b420787a568b1443" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42309 } }
I20260924 09:58:42.837137 13301 leader_election.cc:302] T 00000000000000000000000000000000 P c7230627fb3d4044b420787a568b1443 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c7230627fb3d4044b420787a568b1443; no voters:
I20260924 09:58:42.837780 13301 leader_election.cc:288] T 00000000000000000000000000000000 P c7230627fb3d4044b420787a568b1443 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 09:58:42.837891 13305 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c7230627fb3d4044b420787a568b1443 [term 2 FOLLOWER]: Leader election won for term 2
I20260924 09:58:42.839388 13305 raft_consensus.cc:686] T 00000000000000000000000000000000 P c7230627fb3d4044b420787a568b1443 [term 2 LEADER]: Becoming Leader. State: Replica: c7230627fb3d4044b420787a568b1443, State: Running, Role: LEADER
I20260924 09:58:42.839828 13305 consensus_queue.cc:237] T 00000000000000000000000000000000 P c7230627fb3d4044b420787a568b1443 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7230627fb3d4044b420787a568b1443" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42309 } }
I20260924 09:58:42.840226 13301 sys_catalog.cc:564] T 00000000000000000000000000000000 P c7230627fb3d4044b420787a568b1443 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 09:58:42.842075 13306 sys_catalog.cc:455] T 00000000000000000000000000000000 P c7230627fb3d4044b420787a568b1443 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "c7230627fb3d4044b420787a568b1443" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7230627fb3d4044b420787a568b1443" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42309 } } }
I20260924 09:58:42.842419 13306 sys_catalog.cc:458] T 00000000000000000000000000000000 P c7230627fb3d4044b420787a568b1443 [sys.catalog]: This master's current role is: LEADER
I20260924 09:58:42.847311 13307 sys_catalog.cc:455] T 00000000000000000000000000000000 P c7230627fb3d4044b420787a568b1443 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c7230627fb3d4044b420787a568b1443. Latest consensus state: current_term: 2 leader_uuid: "c7230627fb3d4044b420787a568b1443" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7230627fb3d4044b420787a568b1443" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42309 } } }
I20260924 09:58:42.847605 13307 sys_catalog.cc:458] T 00000000000000000000000000000000 P c7230627fb3d4044b420787a568b1443 [sys.catalog]: This master's current role is: LEADER
I20260924 09:58:42.848299 13311 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 09:58:42.853556 13311 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 09:58:42.854858 13311 catalog_manager.cc:1259] Loaded cluster ID: e50fa497e20d46fd80e285dab4dbfc82
I20260924 09:58:42.854988 13311 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 09:58:42.858807 13311 catalog_manager.cc:1504] Loading token signing keys...
I20260924 09:58:42.860175 13311 catalog_manager.cc:5879] T 00000000000000000000000000000000 P c7230627fb3d4044b420787a568b1443: Loaded TSK: 0
I20260924 09:58:42.861671 13311 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 09:58:42.864588 13301 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790243735634835-388-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
W20260924 09:58:43.210318 13303 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 09:58:43.210590 13303 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 09:58:43.210752 13303 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 09:58:43.219743 13303 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 09:58:43.219864 13303 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 09:58:43.220016 13303 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260924 09:58:43.234490 13303 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790243735634835-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790243735634835-388-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:44351/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790243735634835-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:40527
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790243735634835-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=44873
--tserver_master_addrs=127.0.97.62:42309
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790243735634835-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 09:58:43.236301 13303 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 09:58:43.238044 13303 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 09:58:43.257928 13338 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:58:43.270677 13339 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:58:43.271423 13342 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 09:58:43.272176 13303 server_base.cc:1034] running on GCE node
I20260924 09:58:43.272862 13303 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 09:58:43.273890 13303 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 09:58:43.275163 13303 hybrid_clock.cc:648] HybridClock initialized: now 1790243923274944 us; error 213 us; skew 500 ppm
I20260924 09:58:43.275511 13303 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 09:58:43 dist-test-slave-w8d2 krb5kdc[12255](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243923, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 09:58:43.278681 13303 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260924 09:58:43.280400 13303 webserver.cc:460] Webserver started at http://127.0.97.1:44873/ using document root <none> and password file <none>
I20260924 09:58:43.281090 13303 fs_manager.cc:359] Metadata directory not provided
I20260924 09:58:43.281224 13303 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 09:58:43 dist-test-slave-w8d2 krb5kdc[12255](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243923, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260924 09:58:43.329383 13303 fs_manager.cc:709] Time spent opening directory manager: real 0.018s user 0.001s sys 0.015s
I20260924 09:58:43.344024 13349 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 09:58:43.347867 13303 fs_manager.cc:725] Time spent opening block manager: real 0.016s user 0.002s sys 0.004s
I20260924 09:58:43.348079 13303 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790243735634835-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790243735634835-388-0/minicluster-data/ts-0/wal
uuid: "6675bb8ea82e4825b85aa9a50d9533d2"
format_stamp: "Formatted at 2026-09-24 09:58:40 on dist-test-slave-w8d2"
server_key: "e3632be09f1faec85388161d61582097"
server_key_iv: "ff04a505ab45fa8806c8519278b3a1f9"
server_key_version: "kuduclusterkey@0"
I20260924 09:58:43.349475 13303 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790243735634835-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790243735634835-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790243735634835-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 09:58:43.426020 13303 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 09:58:43.430586 13303 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 09:58:43.430852 13303 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 09:58:43.431844 13303 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 09:58:43.437685 13303 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 09:58:43.437779 13303 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 09:58:43.437886 13303 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 09:58:43.437963 13303 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 09:58:43.484723 13303 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:40527
I20260924 09:58:43.484782 13462 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:40527 every 8 connection(s)
I20260924 09:58:43.486159 13303 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790243735634835-388-0/minicluster-data/ts-0/data/info.pb
I20260924 09:58:43.492612 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu as pid 13303
I20260924 09:58:43.493435 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
/tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790243735634835-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790243735634835-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790243735634835-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790243735634835-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:44351/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:42709
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:42309
--webserver_port=34931
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790243735634835-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:44351/kms with env {KRB5CCNAME=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790243735634835-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790243735634835-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790243735634835-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 09:58:43 dist-test-slave-w8d2 krb5kdc[12255](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243923, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 09:58:43.558909 13467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:58:43.507892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42309 (local address 127.0.97.1:32929)
0924 09:58:43.526021 (+ 18129us) negotiation.cc:107] Waiting for socket to connect
0924 09:58:43.526096 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0924 09:58:43.535353 (+ 9257us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:58:43.538057 (+ 2704us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:58:43.538070 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:58:43.538727 (+ 657us) client_negotiation.cc:189] Negotiated authn=SASL
0924 09:58:43.539582 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:58:43.539611 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:58:43.541864 (+ 2253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:58:43.541869 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:58:43.543883 (+ 2014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:58:43.543908 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:58:43.545576 (+ 1668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:58:43.545584 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:58:43.545862 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:58:43.546577 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 09:58:43.546609 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0924 09:58:43.549266 (+ 2657us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 09:58:43.553626 (+ 4360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 09:58:43.553634 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 09:58:43.553648 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 09:58:43.554254 (+ 606us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 09:58:43.554900 (+ 646us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 09:58:43.554905 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 09:58:43.554910 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 09:58:43.555156 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 09:58:43.556407 (+ 1251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 09:58:43.556415 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 09:58:43.556777 (+ 362us) client_negotiation.cc:769] Sending connection context
0924 09:58:43.557121 (+ 344us) client_negotiation.cc:240] Negotiation successful
0924 09:58:43.557654 (+ 533us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17747,"thread_start_us":18484,"threads_started":1}
I20260924 09:58:43.561323 13466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:58:43.507817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:32929 (local address 127.0.97.62:42309)
0924 09:58:43.536293 (+ 28476us) server_negotiation.cc:206] Beginning negotiation
0924 09:58:43.536331 (+ 38us) server_negotiation.cc:399] Waiting for connection header
0924 09:58:43.536382 (+ 51us) server_negotiation.cc:407] Connection header received
0924 09:58:43.536842 (+ 460us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:58:43.536865 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:58:43.537386 (+ 521us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:58:43.537829 (+ 443us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:58:43.539826 (+ 1997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:58:43.541658 (+ 1832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:58:43.544081 (+ 2423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:58:43.545025 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:58:43.545216 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:58:43.549588 (+ 4372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:58:43.549599 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:58:43.549618 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:58:43.549669 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:58:43.553424 (+ 3755us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:58:43.554442 (+ 1018us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:58:43.554451 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:58:43.554458 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:58:43.554704 (+ 246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:58:43.555322 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:58:43.555327 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:58:43.555330 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:58:43.556146 (+ 816us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:58:43.556330 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0924 09:58:43.559559 (+ 3229us) server_negotiation.cc:299] Negotiation successful
0924 09:58:43.560002 (+ 443us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28142,"thread_start_us":27915,"threads_started":1}
I20260924 09:58:43.564677 13463 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42309
I20260924 09:58:43.565008 13463 heartbeater.cc:464] Registering TS with master...
I20260924 09:58:43.565883 13463 heartbeater.cc:511] Master 127.0.97.62:42309 requested a full tablet report, sending...
W20260924 09:58:43.569098 13463 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:42309 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260924 09:58:43.569329 13463 heartbeater.cc:464] Registering TS with master...
I20260924 09:58:43.569623 13463 heartbeater.cc:511] Master 127.0.97.62:42309 requested a full tablet report, sending...
I20260924 09:58:43.570870 13463 heartbeater.cc:464] Registering TS with master...
I20260924 09:58:43.571139 13463 heartbeater.cc:511] Master 127.0.97.62:42309 requested a full tablet report, sending...
W20260924 09:58:43.572204 13463 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 09:58:43.876736 13465 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 09:58:43.877095 13465 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 09:58:43.877295 13465 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 09:58:43.888414 13465 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 09:58:43.888595 13465 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 09:58:43.888789 13465 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260924 09:58:43.904563 13465 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790243735634835-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790243735634835-388-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:44351/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790243735634835-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:42709
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790243735634835-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=34931
--tserver_master_addrs=127.0.97.62:42309
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790243735634835-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 09:58:43.906541 13465 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 09:58:43.908217 13465 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 09:58:43.924937 13484 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:58:43.924962 13482 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:58:43.925817 13481 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 09:58:43.926128 13465 server_base.cc:1034] running on GCE node
I20260924 09:58:43.926733 13465 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 09:58:43.927717 13465 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 09:58:43.928946 13465 hybrid_clock.cc:648] HybridClock initialized: now 1790243923928880 us; error 57 us; skew 500 ppm
I20260924 09:58:43.929240 13465 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 09:58:43 dist-test-slave-w8d2 krb5kdc[12255](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243923, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 09:58:43.932255 13465 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260924 09:58:43.936411 13465 webserver.cc:460] Webserver started at http://127.0.97.2:34931/ using document root <none> and password file <none>
I20260924 09:58:43.937071 13465 fs_manager.cc:359] Metadata directory not provided
I20260924 09:58:43.937196 13465 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 09:58:43 dist-test-slave-w8d2 krb5kdc[12255](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243923, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260924 09:58:43.976938 13465 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.002s sys 0.002s
I20260924 09:58:43.995155 13494 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 09:58:43.997282 13465 fs_manager.cc:725] Time spent opening block manager: real 0.019s user 0.007s sys 0.009s
I20260924 09:58:43.997512 13465 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790243735634835-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790243735634835-388-0/minicluster-data/ts-1/wal
uuid: "5053831bb8594125aebee1c74b106a25"
format_stamp: "Formatted at 2026-09-24 09:58:41 on dist-test-slave-w8d2"
server_key: "21f32927fe9187890712941dd6781983"
server_key_iv: "f27ef57b6790efd46e68f3a8ea316c18"
server_key_version: "kuduclusterkey@0"
I20260924 09:58:43.998974 13465 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790243735634835-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790243735634835-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790243735634835-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 09:58:44.016331 13465 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 09:58:44.020200 13465 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 09:58:44.020448 13465 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 09:58:44.021417 13465 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 09:58:44.022809 13465 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 09:58:44.022900 13465 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 09:58:44.023006 13465 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 09:58:44.023075 13465 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 09:58:44 dist-test-slave-w8d2 krb5kdc[12255](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790243924, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 09:58:44.138474 13465 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:42709
I20260924 09:58:44.138564 13606 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:42709 every 8 connection(s)
I20260924 09:58:44.139947 13465 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790243735634835-388-0/minicluster-data/ts-1/data/info.pb
I20260924 09:58:44.150259 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu as pid 13465
I20260924 09:58:44.151028 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
/tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790243735634835-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790243735634835-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790243735634835-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790243735634835-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:44351/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:44923
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:42309
--webserver_port=41599
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790243735634835-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:44351/kms with env {KRB5CCNAME=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790243735634835-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790243735634835-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790243735634835-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 09:58:44 dist-test-slave-w8d2 krb5kdc[12255](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243923, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 09:58:44.167639 13609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:58:44.143033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:34345 (local address 127.0.97.62:42309)
0924 09:58:44.147339 (+ 4306us) server_negotiation.cc:206] Beginning negotiation
0924 09:58:44.147345 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 09:58:44.147903 (+ 558us) server_negotiation.cc:407] Connection header received
0924 09:58:44.150511 (+ 2608us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:58:44.150518 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:58:44.150608 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:58:44.150710 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:58:44.152921 (+ 2211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:58:44.154037 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:58:44.157983 (+ 3946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:58:44.158468 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:58:44.158608 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:58:44.162033 (+ 3425us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:58:44.162037 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:58:44.162041 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:58:44.162074 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:58:44.164700 (+ 2626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:58:44.165385 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:58:44.165390 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:58:44.165394 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:58:44.165523 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:58:44.165937 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:58:44.165941 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:58:44.165945 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:58:44.166201 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:58:44.166327 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0924 09:58:44.167037 (+ 710us) server_negotiation.cc:299] Negotiation successful
0924 09:58:44.167286 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4163,"thread_start_us":4017,"threads_started":1}
I20260924 09:58:44.168781 13610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:58:44.143651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42309 (local address 127.0.97.2:34345)
0924 09:58:44.147719 (+ 4068us) negotiation.cc:107] Waiting for socket to connect
0924 09:58:44.147770 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0924 09:58:44.150066 (+ 2296us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:58:44.151161 (+ 1095us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:58:44.151174 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:58:44.151839 (+ 665us) client_negotiation.cc:189] Negotiated authn=SASL
0924 09:58:44.152704 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:58:44.152729 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:58:44.154196 (+ 1467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:58:44.154202 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:58:44.157786 (+ 3584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:58:44.157802 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:58:44.158931 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:58:44.158953 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:58:44.159151 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:58:44.159796 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 09:58:44.159819 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 09:58:44.161772 (+ 1953us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 09:58:44.164834 (+ 3062us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 09:58:44.164843 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 09:58:44.164862 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0924 09:58:44.165264 (+ 402us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 09:58:44.165622 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 09:58:44.165627 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 09:58:44.165631 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 09:58:44.165825 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 09:58:44.166324 (+ 499us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 09:58:44.166332 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 09:58:44.166642 (+ 310us) client_negotiation.cc:769] Sending connection context
0924 09:58:44.166920 (+ 278us) client_negotiation.cc:240] Negotiation successful
0924 09:58:44.167523 (+ 603us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3771,"thread_start_us":3559,"threads_started":1}
I20260924 09:58:44.170174 13607 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42309
I20260924 09:58:44.170454 13607 heartbeater.cc:464] Registering TS with master...
I20260924 09:58:44.171204 13607 heartbeater.cc:511] Master 127.0.97.62:42309 requested a full tablet report, sending...
I20260924 09:58:44.173295 13266 ts_manager.cc:194] Registered new tserver with Master: 5053831bb8594125aebee1c74b106a25 (127.0.97.2:42709)
I20260924 09:58:44.174830 13266 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:34345
Sep 24 09:58:44 dist-test-slave-w8d2 krb5kdc[12255](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243924, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260924 09:58:44.572342 13463 heartbeater.cc:464] Registering TS with master...
I20260924 09:58:44.572767 13463 heartbeater.cc:511] Master 127.0.97.62:42309 requested a full tablet report, sending...
I20260924 09:58:44.573966 13266 ts_manager.cc:194] Registered new tserver with Master: 6675bb8ea82e4825b85aa9a50d9533d2 (127.0.97.1:40527)
I20260924 09:58:44.575031 13266 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:32929
W20260924 09:58:44.599766 13611 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 09:58:44.600133 13611 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 09:58:44.600267 13611 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 09:58:44.610615 13611 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 09:58:44.610764 13611 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 09:58:44.610951 13611 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260924 09:58:44.624931 13611 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790243735634835-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790243735634835-388-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:44351/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790243735634835-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:44923
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790243735634835-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=41599
--tserver_master_addrs=127.0.97.62:42309
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790243735634835-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 09:58:44.626564 13611 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 09:58:44.628161 13611 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 09:58:44.667758 13617 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:58:44.679471 13618 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:58:44.679497 13621 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 09:58:44.679729 13611 server_base.cc:1034] running on GCE node
I20260924 09:58:44.680608 13611 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 09:58:44.692237 13611 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 09:58:44.693482 13611 hybrid_clock.cc:648] HybridClock initialized: now 1790243924693410 us; error 71 us; skew 500 ppm
I20260924 09:58:44.693862 13611 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 09:58:44 dist-test-slave-w8d2 krb5kdc[12255](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243924, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 09:58:44.701153 13611 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260924 09:58:44.721815 13611 webserver.cc:460] Webserver started at http://127.0.97.3:41599/ using document root <none> and password file <none>
I20260924 09:58:44.722944 13611 fs_manager.cc:359] Metadata directory not provided
I20260924 09:58:44.723165 13611 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 09:58:44 dist-test-slave-w8d2 krb5kdc[12255](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243924, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260924 09:58:44.765189 13611 fs_manager.cc:709] Time spent opening directory manager: real 0.018s user 0.007s sys 0.003s
I20260924 09:58:44.779915 13628 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 09:58:44.782047 13611 fs_manager.cc:725] Time spent opening block manager: real 0.014s user 0.002s sys 0.002s
I20260924 09:58:44.782308 13611 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790243735634835-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790243735634835-388-0/minicluster-data/ts-2/wal
uuid: "ec84e06401aa46209a01c4d6fc74ba79"
format_stamp: "Formatted at 2026-09-24 09:58:41 on dist-test-slave-w8d2"
server_key: "b38eeb2472652491b15eb90fc8adcc38"
server_key_iv: "905018f113d09bca4a4b41385f9c1af1"
server_key_version: "kuduclusterkey@0"
I20260924 09:58:44.783763 13611 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790243735634835-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790243735634835-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790243735634835-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 09:58:44.835563 13611 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 09:58:44.840430 13611 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 09:58:44.840755 13611 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 09:58:44.860896 13611 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
Sep 24 09:58:44 dist-test-slave-w8d2 krb5kdc[12255](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243924, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260924 09:58:44.866506 13611 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 09:58:44.866614 13611 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 09:58:44.866721 13611 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 09:58:44.866793 13611 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 09:58:45.179654 13607 heartbeater.cc:503] Master 127.0.97.62:42309 was elected leader, sending a full tablet report...
I20260924 09:58:45.225333 13611 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:44923
I20260924 09:58:45.225576 13748 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:44923 every 8 connection(s)
I20260924 09:58:45.227103 13611 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790243735634835-388-0/minicluster-data/ts-2/data/info.pb
I20260924 09:58:45.228267 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu as pid 13611
Sep 24 09:58:45 dist-test-slave-w8d2 krb5kdc[12255](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243924, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 09:58:45.267809 13751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:58:45.229261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40440 (local address 127.0.97.62:42309)
0924 09:58:45.251914 (+ 22653us) server_negotiation.cc:206] Beginning negotiation
0924 09:58:45.251920 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 09:58:45.251939 (+ 19us) server_negotiation.cc:407] Connection header received
0924 09:58:45.252012 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:58:45.252017 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:58:45.252120 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:58:45.252315 (+ 195us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:58:45.254296 (+ 1981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:58:45.255119 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:58:45.257030 (+ 1911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:58:45.257419 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:58:45.257523 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:58:45.261156 (+ 3633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:58:45.261161 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:58:45.261164 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:58:45.261198 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:58:45.264822 (+ 3624us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:58:45.265588 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:58:45.265593 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:58:45.265597 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:58:45.265725 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:58:45.266360 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:58:45.266365 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:58:45.266368 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:58:45.266677 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:58:45.266891 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0924 09:58:45.267292 (+ 401us) server_negotiation.cc:299] Negotiation successful
0924 09:58:45.267487 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22553,"thread_start_us":22456,"threads_started":1}
I20260924 09:58:45.274073 13754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:58:45.251899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:38207 (local address 127.0.97.62:42309)
0924 09:58:45.252412 (+ 513us) server_negotiation.cc:206] Beginning negotiation
0924 09:58:45.252418 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 09:58:45.252437 (+ 19us) server_negotiation.cc:407] Connection header received
0924 09:58:45.252486 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:58:45.252490 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:58:45.252560 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:58:45.254531 (+ 1971us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:58:45.254758 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:58:45.256239 (+ 1481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:58:45.258631 (+ 2392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:58:45.259301 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:58:45.260194 (+ 893us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:58:45.265034 (+ 4840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:58:45.265040 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:58:45.265044 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:58:45.265076 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:58:45.270561 (+ 5485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:58:45.271505 (+ 944us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:58:45.271511 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:58:45.271514 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:58:45.271658 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:58:45.272186 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:58:45.272191 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:58:45.272194 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:58:45.272518 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:58:45.272654 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0924 09:58:45.273501 (+ 847us) server_negotiation.cc:299] Negotiation successful
0924 09:58:45.273703 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":396,"thread_start_us":260,"threads_started":1}
I20260924 09:58:45.275163 13753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:58:45.231627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42309 (local address 127.0.97.3:38207)
0924 09:58:45.249479 (+ 17852us) negotiation.cc:107] Waiting for socket to connect
0924 09:58:45.249531 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0924 09:58:45.251204 (+ 1673us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:58:45.252977 (+ 1773us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:58:45.252990 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:58:45.253567 (+ 577us) client_negotiation.cc:189] Negotiated authn=SASL
0924 09:58:45.254361 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:58:45.254385 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:58:45.256484 (+ 2099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:58:45.256489 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:58:45.258434 (+ 1945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:58:45.258456 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:58:45.259520 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:58:45.259527 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:58:45.259798 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:58:45.260476 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 09:58:45.260503 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0924 09:58:45.263325 (+ 2822us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 09:58:45.270712 (+ 7387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 09:58:45.270722 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 09:58:45.270739 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 09:58:45.271378 (+ 639us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 09:58:45.271780 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 09:58:45.271785 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 09:58:45.271788 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 09:58:45.272041 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 09:58:45.272662 (+ 621us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 09:58:45.272670 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 09:58:45.273068 (+ 398us) client_negotiation.cc:769] Sending connection context
0924 09:58:45.273348 (+ 280us) client_negotiation.cc:240] Negotiation successful
0924 09:58:45.273841 (+ 493us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17553,"thread_start_us":16878,"threads_started":1}
I20260924 09:58:45.276531 13749 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42309
I20260924 09:58:45.276863 13749 heartbeater.cc:464] Registering TS with master...
I20260924 09:58:45.277714 13749 heartbeater.cc:511] Master 127.0.97.62:42309 requested a full tablet report, sending...
I20260924 09:58:45.279212 13266 ts_manager.cc:194] Registered new tserver with Master: ec84e06401aa46209a01c4d6fc74ba79 (127.0.97.3:44923)
I20260924 09:58:45.280267 13266 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:38207
I20260924 09:58:45.284909 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 09:58:45.306185 13754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:58:45.289570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40442 (local address 127.0.97.62:42309)
0924 09:58:45.289738 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0924 09:58:45.289745 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 09:58:45.290415 (+ 670us) server_negotiation.cc:407] Connection header received
0924 09:58:45.290564 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:58:45.290569 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:58:45.290657 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:58:45.290756 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:58:45.292594 (+ 1838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:58:45.293658 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:58:45.295489 (+ 1831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:58:45.296103 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:58:45.296232 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:58:45.299180 (+ 2948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:58:45.299186 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:58:45.299191 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:58:45.299228 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:58:45.302666 (+ 3438us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:58:45.303418 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:58:45.303423 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:58:45.303427 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:58:45.303593 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:58:45.304124 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:58:45.304129 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:58:45.304133 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:58:45.304485 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:58:45.304719 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0924 09:58:45.305624 (+ 905us) server_negotiation.cc:299] Negotiation successful
0924 09:58:45.305837 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 09:58:45.312410 13266 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40442:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 09:58:45.323434 13266 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 09:58:45.370661 13766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:58:45.342423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40527 (local address 127.0.0.1:47692)
0924 09:58:45.360745 (+ 18322us) negotiation.cc:107] Waiting for socket to connect
0924 09:58:45.360767 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 09:58:45.360921 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:58:45.362417 (+ 1496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:58:45.362422 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:58:45.362459 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 09:58:45.362755 (+ 296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:58:45.362766 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:58:45.364777 (+ 2011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:58:45.364782 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:58:45.367394 (+ 2612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:58:45.367409 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:58:45.369586 (+ 2177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:58:45.369593 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:58:45.369800 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:58:45.369850 (+ 50us) client_negotiation.cc:769] Sending connection context
0924 09:58:45.369984 (+ 134us) client_negotiation.cc:240] Negotiation successful
0924 09:58:45.370172 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18189,"mutex_wait_us":1,"thread_start_us":12684,"threads_started":1}
I20260924 09:58:45.371726 13768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:58:45.342578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47692 (local address 127.0.97.1:40527)
0924 09:58:45.358950 (+ 16372us) server_negotiation.cc:206] Beginning negotiation
0924 09:58:45.358960 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 09:58:45.361059 (+ 2099us) server_negotiation.cc:407] Connection header received
0924 09:58:45.361166 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:58:45.361174 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:58:45.361395 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:58:45.361602 (+ 207us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 09:58:45.362948 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:58:45.364605 (+ 1657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:58:45.367586 (+ 2981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:58:45.369063 (+ 1477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:58:45.369182 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:58:45.369420 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0924 09:58:45.371047 (+ 1627us) server_negotiation.cc:299] Negotiation successful
0924 09:58:45.371218 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16171,"thread_start_us":16388,"threads_started":1}
I20260924 09:58:45.371850 13767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:58:45.341896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:44923 (local address 127.0.0.1:60764)
0924 09:58:45.354821 (+ 12925us) negotiation.cc:107] Waiting for socket to connect
0924 09:58:45.354847 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0924 09:58:45.355023 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:58:45.361950 (+ 6927us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:58:45.361956 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:58:45.361983 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 09:58:45.362318 (+ 335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:58:45.362330 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:58:45.365996 (+ 3666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:58:45.366001 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:58:45.368086 (+ 2085us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:58:45.368101 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:58:45.371064 (+ 2963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:58:45.371071 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:58:45.371262 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:58:45.371297 (+ 35us) client_negotiation.cc:769] Sending connection context
0924 09:58:45.371362 (+ 65us) client_negotiation.cc:240] Negotiation successful
0924 09:58:45.371496 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12764,"mutex_wait_us":25,"thread_start_us":18560,"threads_started":1}
I20260924 09:58:45.373108 13769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:58:45.342822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60764 (local address 127.0.97.3:44923)
0924 09:58:45.359619 (+ 16797us) server_negotiation.cc:206] Beginning negotiation
0924 09:58:45.359628 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 09:58:45.359661 (+ 33us) server_negotiation.cc:407] Connection header received
0924 09:58:45.359776 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:58:45.359785 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:58:45.360098 (+ 313us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:58:45.362119 (+ 2021us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 09:58:45.362551 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:58:45.365844 (+ 3293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:58:45.368290 (+ 2446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:58:45.370846 (+ 2556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:58:45.371974 (+ 1128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:58:45.372550 (+ 576us) server_negotiation.cc:1035] Waiting for connection context
0924 09:58:45.372620 (+ 70us) server_negotiation.cc:299] Negotiation successful
0924 09:58:45.372776 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16633,"thread_start_us":16497,"threads_started":1}
I20260924 09:58:45.376497 13398 tablet_service.cc:1467] Processing CreateTablet for tablet 58852f9541134704b8c4485490aa1f31 (DEFAULT_TABLE table=test-table [id=ebe531ac9a574c57924e73a240490b18]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 09:58:45.376513 13677 tablet_service.cc:1467] Processing CreateTablet for tablet 58852f9541134704b8c4485490aa1f31 (DEFAULT_TABLE table=test-table [id=ebe531ac9a574c57924e73a240490b18]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 09:58:45.377918 13677 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 58852f9541134704b8c4485490aa1f31. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 09:58:45.377936 13398 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 58852f9541134704b8c4485490aa1f31. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 09:58:45.385841 13764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:58:45.340036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:42709 (local address 127.0.0.1:40840)
0924 09:58:45.340877 (+ 841us) negotiation.cc:107] Waiting for socket to connect
0924 09:58:45.340920 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 09:58:45.341120 (+ 200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:58:45.361722 (+ 20602us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:58:45.361729 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:58:45.361757 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 09:58:45.364987 (+ 3230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:58:45.365001 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:58:45.366789 (+ 1788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:58:45.366795 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:58:45.376272 (+ 9477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:58:45.376288 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:58:45.385032 (+ 8744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:58:45.385040 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:58:45.385212 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:58:45.385246 (+ 34us) client_negotiation.cc:769] Sending connection context
0924 09:58:45.385317 (+ 71us) client_negotiation.cc:240] Negotiation successful
0924 09:58:45.385469 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":506,"mutex_wait_us":2976,"thread_start_us":346,"threads_started":1}
I20260924 09:58:45.386946 13765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:58:45.340269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40840 (local address 127.0.97.2:42709)
0924 09:58:45.360989 (+ 20720us) server_negotiation.cc:206] Beginning negotiation
0924 09:58:45.360996 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 09:58:45.361027 (+ 31us) server_negotiation.cc:407] Connection header received
0924 09:58:45.361119 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:58:45.361126 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:58:45.361369 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:58:45.361565 (+ 196us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 09:58:45.365177 (+ 3612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:58:45.366622 (+ 1445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:58:45.383334 (+ 16712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:58:45.384777 (+ 1443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:58:45.386191 (+ 1414us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:58:45.386416 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0924 09:58:45.386490 (+ 74us) server_negotiation.cc:299] Negotiation successful
0924 09:58:45.386638 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20538,"thread_start_us":21490,"threads_started":1}
I20260924 09:58:45.390525 13542 tablet_service.cc:1467] Processing CreateTablet for tablet 58852f9541134704b8c4485490aa1f31 (DEFAULT_TABLE table=test-table [id=ebe531ac9a574c57924e73a240490b18]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 09:58:45.391924 13542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 58852f9541134704b8c4485490aa1f31. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 09:58:45.399142 13770 tablet_bootstrap.cc:492] T 58852f9541134704b8c4485490aa1f31 P ec84e06401aa46209a01c4d6fc74ba79: Bootstrap starting.
I20260924 09:58:45.402204 13770 tablet_bootstrap.cc:654] T 58852f9541134704b8c4485490aa1f31 P ec84e06401aa46209a01c4d6fc74ba79: Neither blocks nor log segments found. Creating new log.
I20260924 09:58:45.403523 13770 log.cc:824] T 58852f9541134704b8c4485490aa1f31 P ec84e06401aa46209a01c4d6fc74ba79: Log is configured to *not* fsync() on all Append() calls
I20260924 09:58:45.403855 13771 tablet_bootstrap.cc:492] T 58852f9541134704b8c4485490aa1f31 P 6675bb8ea82e4825b85aa9a50d9533d2: Bootstrap starting.
I20260924 09:58:45.410959 13771 tablet_bootstrap.cc:654] T 58852f9541134704b8c4485490aa1f31 P 6675bb8ea82e4825b85aa9a50d9533d2: Neither blocks nor log segments found. Creating new log.
I20260924 09:58:45.412191 13771 log.cc:824] T 58852f9541134704b8c4485490aa1f31 P 6675bb8ea82e4825b85aa9a50d9533d2: Log is configured to *not* fsync() on all Append() calls
I20260924 09:58:45.412906 13770 tablet_bootstrap.cc:492] T 58852f9541134704b8c4485490aa1f31 P ec84e06401aa46209a01c4d6fc74ba79: No bootstrap required, opened a new log
I20260924 09:58:45.413215 13770 ts_tablet_manager.cc:1397] T 58852f9541134704b8c4485490aa1f31 P ec84e06401aa46209a01c4d6fc74ba79: Time spent bootstrapping tablet: real 0.014s user 0.009s sys 0.004s
I20260924 09:58:45.424674 13770 raft_consensus.cc:348] T 58852f9541134704b8c4485490aa1f31 P ec84e06401aa46209a01c4d6fc74ba79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5053831bb8594125aebee1c74b106a25" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42709 } } peers { permanent_uuid: "ec84e06401aa46209a01c4d6fc74ba79" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44923 } } peers { permanent_uuid: "6675bb8ea82e4825b85aa9a50d9533d2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40527 } }
I20260924 09:58:45.425206 13770 raft_consensus.cc:374] T 58852f9541134704b8c4485490aa1f31 P ec84e06401aa46209a01c4d6fc74ba79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 09:58:45.425302 13770 raft_consensus.cc:729] T 58852f9541134704b8c4485490aa1f31 P ec84e06401aa46209a01c4d6fc74ba79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec84e06401aa46209a01c4d6fc74ba79, State: Initialized, Role: FOLLOWER
I20260924 09:58:45.426458 13774 tablet_bootstrap.cc:492] T 58852f9541134704b8c4485490aa1f31 P 5053831bb8594125aebee1c74b106a25: Bootstrap starting.
I20260924 09:58:45.430377 13774 tablet_bootstrap.cc:654] T 58852f9541134704b8c4485490aa1f31 P 5053831bb8594125aebee1c74b106a25: Neither blocks nor log segments found. Creating new log.
I20260924 09:58:45.431805 13774 log.cc:824] T 58852f9541134704b8c4485490aa1f31 P 5053831bb8594125aebee1c74b106a25: Log is configured to *not* fsync() on all Append() calls
I20260924 09:58:45.433101 13770 consensus_queue.cc:260] T 58852f9541134704b8c4485490aa1f31 P ec84e06401aa46209a01c4d6fc74ba79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5053831bb8594125aebee1c74b106a25" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42709 } } peers { permanent_uuid: "ec84e06401aa46209a01c4d6fc74ba79" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44923 } } peers { permanent_uuid: "6675bb8ea82e4825b85aa9a50d9533d2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40527 } }
I20260924 09:58:45.435280 13771 tablet_bootstrap.cc:492] T 58852f9541134704b8c4485490aa1f31 P 6675bb8ea82e4825b85aa9a50d9533d2: No bootstrap required, opened a new log
I20260924 09:58:45.435575 13771 ts_tablet_manager.cc:1397] T 58852f9541134704b8c4485490aa1f31 P 6675bb8ea82e4825b85aa9a50d9533d2: Time spent bootstrapping tablet: real 0.032s user 0.016s sys 0.010s
I20260924 09:58:45.438319 13774 tablet_bootstrap.cc:492] T 58852f9541134704b8c4485490aa1f31 P 5053831bb8594125aebee1c74b106a25: No bootstrap required, opened a new log
I20260924 09:58:45.438570 13774 ts_tablet_manager.cc:1397] T 58852f9541134704b8c4485490aa1f31 P 5053831bb8594125aebee1c74b106a25: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.000s
I20260924 09:58:45.444304 13749 heartbeater.cc:503] Master 127.0.97.62:42309 was elected leader, sending a full tablet report...
I20260924 09:58:45.445328 13770 ts_tablet_manager.cc:1428] T 58852f9541134704b8c4485490aa1f31 P ec84e06401aa46209a01c4d6fc74ba79: Time spent starting tablet: real 0.032s user 0.012s sys 0.005s
I20260924 09:58:45.446159 13771 raft_consensus.cc:348] T 58852f9541134704b8c4485490aa1f31 P 6675bb8ea82e4825b85aa9a50d9533d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5053831bb8594125aebee1c74b106a25" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42709 } } peers { permanent_uuid: "ec84e06401aa46209a01c4d6fc74ba79" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44923 } } peers { permanent_uuid: "6675bb8ea82e4825b85aa9a50d9533d2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40527 } }
I20260924 09:58:45.446590 13771 raft_consensus.cc:374] T 58852f9541134704b8c4485490aa1f31 P 6675bb8ea82e4825b85aa9a50d9533d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 09:58:45.446694 13771 raft_consensus.cc:729] T 58852f9541134704b8c4485490aa1f31 P 6675bb8ea82e4825b85aa9a50d9533d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6675bb8ea82e4825b85aa9a50d9533d2, State: Initialized, Role: FOLLOWER
I20260924 09:58:45.447386 13771 consensus_queue.cc:260] T 58852f9541134704b8c4485490aa1f31 P 6675bb8ea82e4825b85aa9a50d9533d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5053831bb8594125aebee1c74b106a25" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42709 } } peers { permanent_uuid: "ec84e06401aa46209a01c4d6fc74ba79" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44923 } } peers { permanent_uuid: "6675bb8ea82e4825b85aa9a50d9533d2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40527 } }
I20260924 09:58:45.452167 13774 raft_consensus.cc:348] T 58852f9541134704b8c4485490aa1f31 P 5053831bb8594125aebee1c74b106a25 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5053831bb8594125aebee1c74b106a25" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42709 } } peers { permanent_uuid: "ec84e06401aa46209a01c4d6fc74ba79" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44923 } } peers { permanent_uuid: "6675bb8ea82e4825b85aa9a50d9533d2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40527 } }
I20260924 09:58:45.452653 13774 raft_consensus.cc:374] T 58852f9541134704b8c4485490aa1f31 P 5053831bb8594125aebee1c74b106a25 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 09:58:45.452754 13774 raft_consensus.cc:729] T 58852f9541134704b8c4485490aa1f31 P 5053831bb8594125aebee1c74b106a25 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5053831bb8594125aebee1c74b106a25, State: Initialized, Role: FOLLOWER
I20260924 09:58:45.453490 13774 consensus_queue.cc:260] T 58852f9541134704b8c4485490aa1f31 P 5053831bb8594125aebee1c74b106a25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5053831bb8594125aebee1c74b106a25" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42709 } } peers { permanent_uuid: "ec84e06401aa46209a01c4d6fc74ba79" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44923 } } peers { permanent_uuid: "6675bb8ea82e4825b85aa9a50d9533d2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40527 } }
I20260924 09:58:45.464015 13463 heartbeater.cc:503] Master 127.0.97.62:42309 was elected leader, sending a full tablet report...
I20260924 09:58:45.464861 13771 ts_tablet_manager.cc:1428] T 58852f9541134704b8c4485490aa1f31 P 6675bb8ea82e4825b85aa9a50d9533d2: Time spent starting tablet: real 0.029s user 0.013s sys 0.015s
I20260924 09:58:45.469264 13774 ts_tablet_manager.cc:1428] T 58852f9541134704b8c4485490aa1f31 P 5053831bb8594125aebee1c74b106a25: Time spent starting tablet: real 0.031s user 0.014s sys 0.014s
W20260924 09:58:45.499037 13750 tablet.cc:2369] T 58852f9541134704b8c4485490aa1f31 P ec84e06401aa46209a01c4d6fc74ba79: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 09:58:45.506762 13464 tablet.cc:2369] T 58852f9541134704b8c4485490aa1f31 P 6675bb8ea82e4825b85aa9a50d9533d2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 09:58:45.639775 13777 raft_consensus.cc:482] T 58852f9541134704b8c4485490aa1f31 P 6675bb8ea82e4825b85aa9a50d9533d2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 09:58:45.640076 13777 raft_consensus.cc:504] T 58852f9541134704b8c4485490aa1f31 P 6675bb8ea82e4825b85aa9a50d9533d2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5053831bb8594125aebee1c74b106a25" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42709 } } peers { permanent_uuid: "ec84e06401aa46209a01c4d6fc74ba79" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44923 } } peers { permanent_uuid: "6675bb8ea82e4825b85aa9a50d9533d2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40527 } }
I20260924 09:58:45.641793 13777 leader_election.cc:288] T 58852f9541134704b8c4485490aa1f31 P 6675bb8ea82e4825b85aa9a50d9533d2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5053831bb8594125aebee1c74b106a25 (127.0.97.2:42709), ec84e06401aa46209a01c4d6fc74ba79 (127.0.97.3:44923)
W20260924 09:58:45.642369 13608 tablet.cc:2369] T 58852f9541134704b8c4485490aa1f31 P 5053831bb8594125aebee1c74b106a25: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 09:58:45.651592 13769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:58:45.643501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:47777 (local address 127.0.97.3:44923)
0924 09:58:45.643701 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0924 09:58:45.643707 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 09:58:45.644892 (+ 1185us) server_negotiation.cc:407] Connection header received
0924 09:58:45.645200 (+ 308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:58:45.645206 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:58:45.645286 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:58:45.645390 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 09:58:45.645892 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:58:45.646886 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:58:45.649417 (+ 2531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:58:45.650493 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:58:45.650611 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:58:45.650713 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0924 09:58:45.651204 (+ 491us) server_negotiation.cc:299] Negotiation successful
0924 09:58:45.651353 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260924 09:58:45.652326 13780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:58:45.642120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:44923 (local address 127.0.97.1:47777)
0924 09:58:45.644775 (+ 2655us) negotiation.cc:107] Waiting for socket to connect
0924 09:58:45.644797 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 09:58:45.645025 (+ 228us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:58:45.645401 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:58:45.645406 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:58:45.645428 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 09:58:45.645764 (+ 336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:58:45.645774 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:58:45.647029 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:58:45.647034 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:58:45.649247 (+ 2213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:58:45.649261 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:58:45.650848 (+ 1587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:58:45.650854 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:58:45.651019 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:58:45.651052 (+ 33us) client_negotiation.cc:769] Sending connection context
0924 09:58:45.651817 (+ 765us) client_negotiation.cc:240] Negotiation successful
0924 09:58:45.651945 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1800,"mutex_wait_us":66,"thread_start_us":2723,"threads_started":1}
I20260924 09:58:45.653380 13765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:58:45.643502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:37801 (local address 127.0.97.2:42709)
0924 09:58:45.643702 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0924 09:58:45.643708 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 09:58:45.645217 (+ 1509us) server_negotiation.cc:407] Connection header received
0924 09:58:45.645351 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:58:45.645356 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:58:45.645439 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:58:45.645523 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 09:58:45.646179 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:58:45.647123 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:58:45.649910 (+ 2787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:58:45.652377 (+ 2467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:58:45.652499 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:58:45.652592 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0924 09:58:45.653030 (+ 438us) server_negotiation.cc:299] Negotiation successful
0924 09:58:45.653152 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 09:58:45.653435 13779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:58:45.642180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:42709 (local address 127.0.97.1:37801)
0924 09:58:45.645117 (+ 2937us) negotiation.cc:107] Waiting for socket to connect
0924 09:58:45.645136 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 09:58:45.645239 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:58:45.645594 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:58:45.645598 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:58:45.645624 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 09:58:45.646020 (+ 396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:58:45.646031 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:58:45.647264 (+ 1233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:58:45.647270 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:58:45.648788 (+ 1518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:58:45.648799 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:58:45.652733 (+ 3934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:58:45.652739 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:58:45.652892 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:58:45.652914 (+ 22us) client_negotiation.cc:769] Sending connection context
0924 09:58:45.652991 (+ 77us) client_negotiation.cc:240] Negotiation successful
0924 09:58:45.653103 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2829,"mutex_wait_us":21,"thread_start_us":1565,"threads_started":1}
I20260924 09:58:45.653573 13704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58852f9541134704b8c4485490aa1f31" candidate_uuid: "6675bb8ea82e4825b85aa9a50d9533d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec84e06401aa46209a01c4d6fc74ba79" is_pre_election: true
I20260924 09:58:45.654166 13704 raft_consensus.cc:2454] T 58852f9541134704b8c4485490aa1f31 P ec84e06401aa46209a01c4d6fc74ba79 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6675bb8ea82e4825b85aa9a50d9533d2 in term 0.
I20260924 09:58:45.654320 13562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58852f9541134704b8c4485490aa1f31" candidate_uuid: "6675bb8ea82e4825b85aa9a50d9533d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5053831bb8594125aebee1c74b106a25" is_pre_election: true
I20260924 09:58:45.654763 13562 raft_consensus.cc:2454] T 58852f9541134704b8c4485490aa1f31 P 5053831bb8594125aebee1c74b106a25 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6675bb8ea82e4825b85aa9a50d9533d2 in term 0.
I20260924 09:58:45.655310 13353 leader_election.cc:302] T 58852f9541134704b8c4485490aa1f31 P 6675bb8ea82e4825b85aa9a50d9533d2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6675bb8ea82e4825b85aa9a50d9533d2, ec84e06401aa46209a01c4d6fc74ba79; no voters:
I20260924 09:58:45.655730 13777 raft_consensus.cc:2781] T 58852f9541134704b8c4485490aa1f31 P 6675bb8ea82e4825b85aa9a50d9533d2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 09:58:45.655853 13777 raft_consensus.cc:482] T 58852f9541134704b8c4485490aa1f31 P 6675bb8ea82e4825b85aa9a50d9533d2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 09:58:45.655948 13777 raft_consensus.cc:3037] T 58852f9541134704b8c4485490aa1f31 P 6675bb8ea82e4825b85aa9a50d9533d2 [term 0 FOLLOWER]: Advancing to term 1
I20260924 09:58:45.658017 13777 raft_consensus.cc:504] T 58852f9541134704b8c4485490aa1f31 P 6675bb8ea82e4825b85aa9a50d9533d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5053831bb8594125aebee1c74b106a25" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42709 } } peers { permanent_uuid: "ec84e06401aa46209a01c4d6fc74ba79" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44923 } } peers { permanent_uuid: "6675bb8ea82e4825b85aa9a50d9533d2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40527 } }
I20260924 09:58:45.658564 13777 leader_election.cc:288] T 58852f9541134704b8c4485490aa1f31 P 6675bb8ea82e4825b85aa9a50d9533d2 [CANDIDATE]: Term 1 election: Requested vote from peers 5053831bb8594125aebee1c74b106a25 (127.0.97.2:42709), ec84e06401aa46209a01c4d6fc74ba79 (127.0.97.3:44923)
I20260924 09:58:45.658981 13562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58852f9541134704b8c4485490aa1f31" candidate_uuid: "6675bb8ea82e4825b85aa9a50d9533d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5053831bb8594125aebee1c74b106a25"
I20260924 09:58:45.659176 13562 raft_consensus.cc:3037] T 58852f9541134704b8c4485490aa1f31 P 5053831bb8594125aebee1c74b106a25 [term 0 FOLLOWER]: Advancing to term 1
I20260924 09:58:45.659413 13704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58852f9541134704b8c4485490aa1f31" candidate_uuid: "6675bb8ea82e4825b85aa9a50d9533d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec84e06401aa46209a01c4d6fc74ba79"
I20260924 09:58:45.659575 13704 raft_consensus.cc:3037] T 58852f9541134704b8c4485490aa1f31 P ec84e06401aa46209a01c4d6fc74ba79 [term 0 FOLLOWER]: Advancing to term 1
I20260924 09:58:45.661536 13704 raft_consensus.cc:2454] T 58852f9541134704b8c4485490aa1f31 P ec84e06401aa46209a01c4d6fc74ba79 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6675bb8ea82e4825b85aa9a50d9533d2 in term 1.
I20260924 09:58:45.661857 13353 leader_election.cc:302] T 58852f9541134704b8c4485490aa1f31 P 6675bb8ea82e4825b85aa9a50d9533d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6675bb8ea82e4825b85aa9a50d9533d2, ec84e06401aa46209a01c4d6fc74ba79; no voters:
I20260924 09:58:45.662098 13777 raft_consensus.cc:2781] T 58852f9541134704b8c4485490aa1f31 P 6675bb8ea82e4825b85aa9a50d9533d2 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 09:58:45.662245 13562 raft_consensus.cc:2454] T 58852f9541134704b8c4485490aa1f31 P 5053831bb8594125aebee1c74b106a25 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6675bb8ea82e4825b85aa9a50d9533d2 in term 1.
I20260924 09:58:45.662685 13777 raft_consensus.cc:686] T 58852f9541134704b8c4485490aa1f31 P 6675bb8ea82e4825b85aa9a50d9533d2 [term 1 LEADER]: Becoming Leader. State: Replica: 6675bb8ea82e4825b85aa9a50d9533d2, State: Running, Role: LEADER
I20260924 09:58:45.663133 13777 consensus_queue.cc:237] T 58852f9541134704b8c4485490aa1f31 P 6675bb8ea82e4825b85aa9a50d9533d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5053831bb8594125aebee1c74b106a25" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42709 } } peers { permanent_uuid: "ec84e06401aa46209a01c4d6fc74ba79" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44923 } } peers { permanent_uuid: "6675bb8ea82e4825b85aa9a50d9533d2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40527 } }
I20260924 09:58:45.667762 13266 catalog_manager.cc:5495] T 58852f9541134704b8c4485490aa1f31 P 6675bb8ea82e4825b85aa9a50d9533d2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6675bb8ea82e4825b85aa9a50d9533d2 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "6675bb8ea82e4825b85aa9a50d9533d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5053831bb8594125aebee1c74b106a25" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42709 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec84e06401aa46209a01c4d6fc74ba79" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44923 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6675bb8ea82e4825b85aa9a50d9533d2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40527 } health_report { overall_health: HEALTHY } } }
I20260924 09:58:45.736086 13768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:58:45.730960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47696 (local address 127.0.97.1:40527)
0924 09:58:45.731137 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0924 09:58:45.731142 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 09:58:45.731159 (+ 17us) server_negotiation.cc:407] Connection header received
0924 09:58:45.731326 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:58:45.731330 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:58:45.731407 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:58:45.731524 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 09:58:45.731796 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:58:45.732901 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:58:45.734184 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:58:45.734558 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:58:45.734651 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:58:45.735051 (+ 400us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 09:58:45.735591 (+ 540us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 09:58:45.735694 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0924 09:58:45.735764 (+ 70us) server_negotiation.cc:299] Negotiation successful
0924 09:58:45.735826 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 09:58:45.743938 13704 raft_consensus.cc:1261] T 58852f9541134704b8c4485490aa1f31 P ec84e06401aa46209a01c4d6fc74ba79 [term 1 FOLLOWER]: Refusing update from remote peer 6675bb8ea82e4825b85aa9a50d9533d2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 09:58:45.743938 13562 raft_consensus.cc:1261] T 58852f9541134704b8c4485490aa1f31 P 5053831bb8594125aebee1c74b106a25 [term 1 FOLLOWER]: Refusing update from remote peer 6675bb8ea82e4825b85aa9a50d9533d2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 09:58:45.744877 13781 consensus_queue.cc:1035] T 58852f9541134704b8c4485490aa1f31 P 6675bb8ea82e4825b85aa9a50d9533d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5053831bb8594125aebee1c74b106a25" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42709 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 09:58:45.745199 13777 consensus_queue.cc:1035] T 58852f9541134704b8c4485490aa1f31 P 6675bb8ea82e4825b85aa9a50d9533d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec84e06401aa46209a01c4d6fc74ba79" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 09:58:45.769846 13266 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40442:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 09:58:45.770121 13266 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40442:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 09:58:45.773775 13266 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c7230627fb3d4044b420787a568b1443: Sending DeleteTablet for 3 replicas of tablet 58852f9541134704b8c4485490aa1f31
I20260924 09:58:45.774777 13542 tablet_service.cc:1514] Processing DeleteTablet for tablet 58852f9541134704b8c4485490aa1f31 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 09:58:45 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:40840
I20260924 09:58:45.774777 13677 tablet_service.cc:1514] Processing DeleteTablet for tablet 58852f9541134704b8c4485490aa1f31 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 09:58:45 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:60764
I20260924 09:58:45.777606 13398 tablet_service.cc:1514] Processing DeleteTablet for tablet 58852f9541134704b8c4485490aa1f31 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 09:58:45 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:47692
I20260924 09:58:45.783495 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu pid 13303
I20260924 09:58:45.798054 13795 tablet_replica.cc:330] T 58852f9541134704b8c4485490aa1f31 P 6675bb8ea82e4825b85aa9a50d9533d2: stopping tablet replica
I20260924 09:58:45.798563 13795 raft_consensus.cc:2229] T 58852f9541134704b8c4485490aa1f31 P 6675bb8ea82e4825b85aa9a50d9533d2 [term 1 LEADER]: Raft consensus shutting down.
I20260924 09:58:45.799258 13795 raft_consensus.cc:2258] T 58852f9541134704b8c4485490aa1f31 P 6675bb8ea82e4825b85aa9a50d9533d2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 09:58:45.801494 13795 ts_tablet_manager.cc:1905] T 58852f9541134704b8c4485490aa1f31 P 6675bb8ea82e4825b85aa9a50d9533d2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 09:58:45.801679 13793 tablet_replica.cc:330] T 58852f9541134704b8c4485490aa1f31 P ec84e06401aa46209a01c4d6fc74ba79: stopping tablet replica
I20260924 09:58:45.802078 13793 raft_consensus.cc:2229] T 58852f9541134704b8c4485490aa1f31 P ec84e06401aa46209a01c4d6fc74ba79 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 09:58:45.802533 13793 raft_consensus.cc:2258] T 58852f9541134704b8c4485490aa1f31 P ec84e06401aa46209a01c4d6fc74ba79 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 09:58:45.803511 13794 tablet_replica.cc:330] T 58852f9541134704b8c4485490aa1f31 P 5053831bb8594125aebee1c74b106a25: stopping tablet replica
I20260924 09:58:45.805177 13793 ts_tablet_manager.cc:1905] T 58852f9541134704b8c4485490aa1f31 P ec84e06401aa46209a01c4d6fc74ba79: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 09:58:45.806802 13794 raft_consensus.cc:2229] T 58852f9541134704b8c4485490aa1f31 P 5053831bb8594125aebee1c74b106a25 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 09:58:45.807319 13794 raft_consensus.cc:2258] T 58852f9541134704b8c4485490aa1f31 P 5053831bb8594125aebee1c74b106a25 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 09:58:45.810000 13795 ts_tablet_manager.cc:1918] T 58852f9541134704b8c4485490aa1f31 P 6675bb8ea82e4825b85aa9a50d9533d2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 09:58:45.810155 13795 log.cc:1196] T 58852f9541134704b8c4485490aa1f31 P 6675bb8ea82e4825b85aa9a50d9533d2: Deleting WAL directory at /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790243735634835-388-0/minicluster-data/ts-0/wal/wals/58852f9541134704b8c4485490aa1f31
I20260924 09:58:45.810663 13795 ts_tablet_manager.cc:1939] T 58852f9541134704b8c4485490aa1f31 P 6675bb8ea82e4825b85aa9a50d9533d2: Deleting consensus metadata
I20260924 09:58:45.812475 13251 catalog_manager.cc:4841] TS 6675bb8ea82e4825b85aa9a50d9533d2 (127.0.97.1:40527): tablet 58852f9541134704b8c4485490aa1f31 (table test-table [id=ebe531ac9a574c57924e73a240490b18]) successfully deleted
I20260924 09:58:45.814404 13794 ts_tablet_manager.cc:1905] T 58852f9541134704b8c4485490aa1f31 P 5053831bb8594125aebee1c74b106a25: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 09:58:45.814563 13793 ts_tablet_manager.cc:1918] T 58852f9541134704b8c4485490aa1f31 P ec84e06401aa46209a01c4d6fc74ba79: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 09:58:45.814733 13793 log.cc:1196] T 58852f9541134704b8c4485490aa1f31 P ec84e06401aa46209a01c4d6fc74ba79: Deleting WAL directory at /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790243735634835-388-0/minicluster-data/ts-2/wal/wals/58852f9541134704b8c4485490aa1f31
I20260924 09:58:45.815405 13793 ts_tablet_manager.cc:1939] T 58852f9541134704b8c4485490aa1f31 P ec84e06401aa46209a01c4d6fc74ba79: Deleting consensus metadata
I20260924 09:58:45.817058 13253 catalog_manager.cc:4841] TS ec84e06401aa46209a01c4d6fc74ba79 (127.0.97.3:44923): tablet 58852f9541134704b8c4485490aa1f31 (table test-table [id=ebe531ac9a574c57924e73a240490b18]) successfully deleted
I20260924 09:58:45.818704 13768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:58:45.784469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47712 (local address 127.0.97.1:40527)
0924 09:58:45.784647 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0924 09:58:45.784653 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 09:58:45.798952 (+ 14299us) server_negotiation.cc:407] Connection header received
0924 09:58:45.799417 (+ 465us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:58:45.799423 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:58:45.799497 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:58:45.799587 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:58:45.802894 (+ 3307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:58:45.804172 (+ 1278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:58:45.806529 (+ 2357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:58:45.807174 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:58:45.807419 (+ 245us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:58:45.810748 (+ 3329us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:58:45.810760 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:58:45.810776 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:58:45.810819 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:58:45.815092 (+ 4273us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:58:45.815995 (+ 903us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:58:45.816002 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:58:45.816006 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:58:45.816180 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:58:45.816619 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:58:45.816624 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:58:45.816627 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:58:45.817389 (+ 762us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:58:45.817573 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0924 09:58:45.818029 (+ 456us) server_negotiation.cc:299] Negotiation successful
0924 09:58:45.818257 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 09:58:45.819300 13458 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 09:58:45.820655 13794 ts_tablet_manager.cc:1918] T 58852f9541134704b8c4485490aa1f31 P 5053831bb8594125aebee1c74b106a25: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 09:58:45.820856 13794 log.cc:1196] T 58852f9541134704b8c4485490aa1f31 P 5053831bb8594125aebee1c74b106a25: Deleting WAL directory at /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790243735634835-388-0/minicluster-data/ts-1/wal/wals/58852f9541134704b8c4485490aa1f31
I20260924 09:58:45.821449 13794 ts_tablet_manager.cc:1939] T 58852f9541134704b8c4485490aa1f31 P 5053831bb8594125aebee1c74b106a25: Deleting consensus metadata
I20260924 09:58:45.823009 13252 catalog_manager.cc:4841] TS 5053831bb8594125aebee1c74b106a25 (127.0.97.2:42709): tablet 58852f9541134704b8c4485490aa1f31 (table test-table [id=ebe531ac9a574c57924e73a240490b18]) successfully deleted
I20260924 09:58:45.915194 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu with pid 13303
I20260924 09:58:45.929471 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu pid 13465
I20260924 09:58:45.941414 13765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:58:45.930190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40850 (local address 127.0.97.2:42709)
0924 09:58:45.930365 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0924 09:58:45.930370 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 09:58:45.930388 (+ 18us) server_negotiation.cc:407] Connection header received
0924 09:58:45.930522 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:58:45.930528 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:58:45.930604 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:58:45.930684 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:58:45.931915 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:58:45.932630 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:58:45.933818 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:58:45.934226 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:58:45.934333 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:58:45.936515 (+ 2182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:58:45.936528 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:58:45.936542 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:58:45.936575 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:58:45.938886 (+ 2311us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:58:45.939535 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:58:45.939543 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:58:45.939548 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:58:45.939691 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:58:45.940028 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:58:45.940033 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:58:45.940035 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:58:45.940334 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:58:45.940635 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0924 09:58:45.940914 (+ 279us) server_negotiation.cc:299] Negotiation successful
0924 09:58:45.941107 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 09:58:45.942126 13602 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 09:58:46.027803 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu with pid 13465
I20260924 09:58:46.040319 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu pid 13611
I20260924 09:58:46.052253 13769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:58:46.041079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60774 (local address 127.0.97.3:44923)
0924 09:58:46.041270 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0924 09:58:46.041276 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 09:58:46.041298 (+ 22us) server_negotiation.cc:407] Connection header received
0924 09:58:46.041413 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:58:46.041417 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:58:46.041479 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:58:46.041555 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:58:46.042797 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:58:46.043547 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:58:46.044693 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:58:46.045068 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:58:46.045153 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:58:46.047557 (+ 2404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:58:46.047572 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:58:46.047588 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:58:46.047624 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:58:46.050121 (+ 2497us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:58:46.050690 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:58:46.050696 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:58:46.050698 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:58:46.050803 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:58:46.051053 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:58:46.051055 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:58:46.051057 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:58:46.051421 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:58:46.051631 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0924 09:58:46.051818 (+ 187us) server_negotiation.cc:299] Negotiation successful
0924 09:58:46.051962 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260924 09:58:46.052848 13744 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 09:58:46.137714 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu with pid 13611
I20260924 09:58:46.150653 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu pid 13233
I20260924 09:58:46.151757 13296 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 09:58:46.260397 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu with pid 13233
I20260924 09:58:46.281363 388 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260924 09:58:46.615926 388 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260924 09:58:46.616133 388 mini_ranger.cc:66] Stopping Ranger...
I20260924 09:58:46.950737 388 mini_ranger.cc:68] Stopped Ranger
2026-09-24 09:58:46.951 UTC [12285] LOG: received smart shutdown request
2026-09-24 09:58:46.954 UTC [12285] LOG: background worker "logical replication launcher" (PID 12294) exited with exit code 1
2026-09-24 09:58:46.954 UTC [12289] LOG: shutting down
2026-09-24 09:58:47.099 UTC [12285] LOG: database system is shut down
2026-09-24T09:58:47Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (50332 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790243735634835-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 09:58:47 dist-test-slave-w8d2 krb5kdc[13809](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 09:58:47 dist-test-slave-w8d2 krb5kdc[13809](info): set up 2 sockets
Sep 24 09:58:47 dist-test-slave-w8d2 krb5kdc[13809](info): commencing operation
krb5kdc: starting...
W20260924 09:58:48.203329 388 mini_kdc.cc:121] Time spent starting KDC: real 0.965s user 0.000s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790243735634835-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790243735634835-388-0/krb5kdc/test-user.keytab.
Sep 24 09:58:48 dist-test-slave-w8d2 krb5kdc[13809](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243928, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T09:58:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T09:58:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790243735634835-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790243735634835-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260924 09:58:48.404769 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
/tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790243735634835-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790243735634835-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790243735634835-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790243735634835-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:32781
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790243735634835-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:32781,127.0.97.61:33167,127.0.97.60:46599
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790243735634835-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790243735634835-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790243735634835-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 09:58:48.716209 13822 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 09:58:48.716496 13822 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 09:58:48.716622 13822 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 09:58:48.723647 13822 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 09:58:48.723750 13822 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 09:58:48.723824 13822 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 09:58:48.723901 13822 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 09:58:48.723960 13822 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 09:58:48.733383 13822 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790243735634835-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790243735634835-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:32781,127.0.97.61:33167,127.0.97.60:46599
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790243735634835-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:32781
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790243735634835-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790243735634835-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 09:58:48.734625 13822 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 09:58:48.735885 13822 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 09:58:48.747347 13827 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:58:48.747392 13828 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:58:48.748807 13830 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 09:58:48.749336 13822 server_base.cc:1034] running on GCE node
I20260924 09:58:48.750142 13822 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 09:58:48.751549 13822 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 09:58:48.752768 13822 hybrid_clock.cc:648] HybridClock initialized: now 1790243928752724 us; error 31 us; skew 500 ppm
I20260924 09:58:48.753096 13822 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 09:58:48 dist-test-slave-w8d2 krb5kdc[13809](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243928, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 09:58:48.755952 13822 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260924 09:58:48.757267 13822 webserver.cc:460] Webserver started at http://127.0.97.62:32999/ using document root <none> and password file <none>
I20260924 09:58:48.757907 13822 fs_manager.cc:359] Metadata directory not provided
I20260924 09:58:48.758000 13822 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 09:58:48.758251 13822 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 09:58:48.761049 13822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790243735634835-388-0/minicluster-data/master-0/data/instance:
uuid: "c1be3585e7dc4a5e837562349eceb0f9"
format_stamp: "Formatted at 2026-09-24 09:58:48 on dist-test-slave-w8d2"
server_key: "668ee1f87ea86e27270fa8432db2e6b0"
server_key_iv: "caef30368c622e597bccdab880c79411"
server_key_version: "encryptionkey@0"
I20260924 09:58:48.761719 13822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790243735634835-388-0/minicluster-data/master-0/wal/instance:
uuid: "c1be3585e7dc4a5e837562349eceb0f9"
format_stamp: "Formatted at 2026-09-24 09:58:48 on dist-test-slave-w8d2"
server_key: "668ee1f87ea86e27270fa8432db2e6b0"
server_key_iv: "caef30368c622e597bccdab880c79411"
server_key_version: "encryptionkey@0"
I20260924 09:58:48.766461 13822 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260924 09:58:48.769869 13837 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 09:58:48.771082 13822 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 09:58:48.771250 13822 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790243735634835-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790243735634835-388-0/minicluster-data/master-0/wal
uuid: "c1be3585e7dc4a5e837562349eceb0f9"
format_stamp: "Formatted at 2026-09-24 09:58:48 on dist-test-slave-w8d2"
server_key: "668ee1f87ea86e27270fa8432db2e6b0"
server_key_iv: "caef30368c622e597bccdab880c79411"
server_key_version: "encryptionkey@0"
I20260924 09:58:48.771442 13822 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790243735634835-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790243735634835-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790243735634835-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 09:58:48.788954 13822 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 09:58:48.792759 13822 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 09:58:48.793002 13822 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 09:58:48.809100 13822 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:32781
I20260924 09:58:48.809131 13888 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:32781 every 8 connection(s)
I20260924 09:58:48.810294 13822 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790243735634835-388-0/minicluster-data/master-0/data/info.pb
I20260924 09:58:48.811756 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu as pid 13822
I20260924 09:58:48.811980 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790243735634835-388-0/minicluster-data/master-0/wal/instance
I20260924 09:58:48.812578 388 external_mini_cluster.cc:1428] Setting key 4ca4cbd25482440d0d2582690798cc9a
I20260924 09:58:48.813827 13889 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 09:58:48.820955 13889 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32781 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.97.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.61@KRBTEST.COM" created.
Sep 24 09:58:48 dist-test-slave-w8d2 krb5kdc[13809](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243928, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 09:58:48.843784 13893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:58:48.824506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32781 (local address 127.0.0.1:38570)
0924 09:58:48.825852 (+ 1346us) negotiation.cc:107] Waiting for socket to connect
0924 09:58:48.825890 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 09:58:48.827644 (+ 1754us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:58:48.828551 (+ 907us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:58:48.828556 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:58:48.829095 (+ 539us) client_negotiation.cc:189] Negotiated authn=SASL
0924 09:58:48.829416 (+ 321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:58:48.829435 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:58:48.831639 (+ 2204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:58:48.831643 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:58:48.832736 (+ 1093us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:58:48.832748 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:58:48.833629 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:58:48.833635 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:58:48.833757 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:58:48.834390 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 09:58:48.834419 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0924 09:58:48.836733 (+ 2314us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 09:58:48.840039 (+ 3306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 09:58:48.840050 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 09:58:48.840062 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0924 09:58:48.840506 (+ 444us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 09:58:48.840924 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 09:58:48.840927 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 09:58:48.840930 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 09:58:48.841085 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 09:58:48.841911 (+ 826us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 09:58:48.841917 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 09:58:48.842104 (+ 187us) client_negotiation.cc:769] Sending connection context
0924 09:58:48.842299 (+ 195us) client_negotiation.cc:240] Negotiation successful
0924 09:58:48.842707 (+ 408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1102,"spinlock_wait_cycles":56192,"thread_start_us":904,"threads_started":1}
I20260924 09:58:48.843815 13892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:58:48.824506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38570 (local address 127.0.97.62:32781)
0924 09:58:48.825321 (+ 815us) server_negotiation.cc:206] Beginning negotiation
0924 09:58:48.825338 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0924 09:58:48.826036 (+ 698us) server_negotiation.cc:407] Connection header received
0924 09:58:48.828005 (+ 1969us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:58:48.828013 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:58:48.828320 (+ 307us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:58:48.828543 (+ 223us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:58:48.829606 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:58:48.831508 (+ 1902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:58:48.832856 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:58:48.833396 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:58:48.833586 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:58:48.836932 (+ 3346us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:58:48.836940 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:58:48.836951 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:58:48.836982 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:58:48.839900 (+ 2918us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:58:48.840625 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:58:48.840634 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:58:48.840637 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:58:48.840814 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:58:48.841183 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:58:48.841188 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:58:48.841191 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:58:48.841771 (+ 580us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:58:48.841959 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0924 09:58:48.842458 (+ 499us) server_negotiation.cc:299] Negotiation successful
0924 09:58:48.842839 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":122,"server-negotiator.queue_time_us":574,"thread_start_us":825,"threads_started":1}
I20260924 09:58:48.846446 13889 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33167 } has no permanent_uuid. Determining permanent_uuid...
I20260924 09:58:48.847522 13893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:58:48.847124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:33167 (local address 127.0.0.1:59478)
0924 09:58:48.847299 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0924 09:58:48.847420 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:33167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260924 09:58:48.847805 13840 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.61:33167: connect: Connection refused (error 111)
I20260924 09:58:48.849323 13893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:58:48.848961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:33167 (local address 127.0.0.1:59484)
0924 09:58:48.849110 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0924 09:58:48.849190 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:33167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260924 09:58:48.849538 13889 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:33167: Network error: Client connection negotiation failed: client connection to 127.0.97.61:33167: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790243735634835-388-0/krb5kdc/kudu_127.0.97.61.keytab.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790243735634835-388-0/krb5kdc/kudu_127.0.97.61.keytab.
I20260924 09:58:48.860402 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
/tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790243735634835-388-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790243735634835-388-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790243735634835-388-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790243735634835-388-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:33167
--webserver_interface=127.0.97.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790243735634835-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:32781,127.0.97.61:33167,127.0.97.60:46599
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790243735634835-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790243735634835-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790243735634835-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 09:58:48.898866 13889 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33167 } attempt: 1
I20260924 09:58:48.899962 13893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:58:48.899539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:33167 (local address 127.0.0.1:59494)
0924 09:58:48.899721 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0924 09:58:48.899807 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:33167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260924 09:58:48.900856 13893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:58:48.900531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:33167 (local address 127.0.0.1:59508)
0924 09:58:48.900667 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0924 09:58:48.900731 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:33167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260924 09:58:48.901043 13889 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:33167: Network error: Client connection negotiation failed: client connection to 127.0.97.61:33167: connect: Connection refused (error 111)
I20260924 09:58:48.969290 13889 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33167 } attempt: 2
I20260924 09:58:48.970495 13893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:58:48.970003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:33167 (local address 127.0.0.1:59518)
0924 09:58:48.970261 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0924 09:58:48.970355 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:33167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260924 09:58:48.971496 13893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:58:48.971109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:33167 (local address 127.0.0.1:59534)
0924 09:58:48.971282 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0924 09:58:48.971368 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:33167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260924 09:58:48.971663 13889 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:33167: Network error: Client connection negotiation failed: client connection to 127.0.97.61:33167: connect: Connection refused (error 111)
I20260924 09:58:49.062891 13889 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33167 } attempt: 3
I20260924 09:58:49.064064 13893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:58:49.063627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:33167 (local address 127.0.0.1:59546)
0924 09:58:49.063820 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0924 09:58:49.063910 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:33167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260924 09:58:49.065039 13893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:58:49.064654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:33167 (local address 127.0.0.1:59558)
0924 09:58:49.064791 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0924 09:58:49.064883 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:33167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260924 09:58:49.065198 13889 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:33167: Network error: Client connection negotiation failed: client connection to 127.0.97.61:33167: connect: Connection refused (error 111)
W20260924 09:58:49.161882 13896 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 09:58:49.162165 13896 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 09:58:49.162297 13896 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 09:58:49.169255 13896 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 09:58:49.169365 13896 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 09:58:49.169445 13896 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 09:58:49.169523 13896 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 09:58:49.169584 13896 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 09:58:49.179137 13896 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790243735634835-388-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790243735634835-388-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:32781,127.0.97.61:33167,127.0.97.60:46599
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790243735634835-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:33167
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790243735634835-388-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.97.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790243735634835-388-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 09:58:49.180435 13896 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 09:58:49.181689 13896 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 09:58:49.192214 13902 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:58:49.192314 13904 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:58:49.192322 13901 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 09:58:49.193181 13896 server_base.cc:1034] running on GCE node
I20260924 09:58:49.193892 13896 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 09:58:49.195106 13896 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 09:58:49.196290 13896 hybrid_clock.cc:648] HybridClock initialized: now 1790243929196233 us; error 47 us; skew 500 ppm
I20260924 09:58:49.196597 13896 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 09:58:49 dist-test-slave-w8d2 krb5kdc[13809](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243929, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 09:58:49.199396 13896 init.cc:377] Logged in from keytab as kudu/127.0.97.61@KRBTEST.COM (short username kudu)
I20260924 09:58:49.200659 13896 webserver.cc:460] Webserver started at http://127.0.97.61:40023/ using document root <none> and password file <none>
I20260924 09:58:49.201267 13896 fs_manager.cc:359] Metadata directory not provided
I20260924 09:58:49.201373 13896 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 09:58:49.201658 13896 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 09:58:49.204428 13896 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790243735634835-388-0/minicluster-data/master-1/data/instance:
uuid: "7eebe7d7bdfb4141aa7e6cbe632d79c5"
format_stamp: "Formatted at 2026-09-24 09:58:49 on dist-test-slave-w8d2"
server_key: "ba16f92977177fec4e0f10545e839d01"
server_key_iv: "c5639825d3ed3511daf4dfdc2126dca8"
server_key_version: "encryptionkey@0"
I20260924 09:58:49.205135 13896 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790243735634835-388-0/minicluster-data/master-1/wal/instance:
uuid: "7eebe7d7bdfb4141aa7e6cbe632d79c5"
format_stamp: "Formatted at 2026-09-24 09:58:49 on dist-test-slave-w8d2"
server_key: "ba16f92977177fec4e0f10545e839d01"
server_key_iv: "c5639825d3ed3511daf4dfdc2126dca8"
server_key_version: "encryptionkey@0"
I20260924 09:58:49.208433 13889 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33167 } attempt: 4
I20260924 09:58:49.209429 13893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:58:49.208965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:33167 (local address 127.0.0.1:59568)
0924 09:58:49.209224 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0924 09:58:49.209303 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:33167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130}
I20260924 09:58:49.209950 13896 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260924 09:58:49.210435 13893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:58:49.210056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:33167 (local address 127.0.0.1:59578)
0924 09:58:49.210198 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0924 09:58:49.210282 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:33167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260924 09:58:49.210630 13889 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:33167: Network error: Client connection negotiation failed: client connection to 127.0.97.61:33167: connect: Connection refused (error 111)
I20260924 09:58:49.213238 13911 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 09:58:49.214505 13896 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 09:58:49.214659 13896 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790243735634835-388-0/minicluster-data/master-1/data,/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790243735634835-388-0/minicluster-data/master-1/wal
uuid: "7eebe7d7bdfb4141aa7e6cbe632d79c5"
format_stamp: "Formatted at 2026-09-24 09:58:49 on dist-test-slave-w8d2"
server_key: "ba16f92977177fec4e0f10545e839d01"
server_key_iv: "c5639825d3ed3511daf4dfdc2126dca8"
server_key_version: "encryptionkey@0"
I20260924 09:58:49.214823 13896 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790243735634835-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790243735634835-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790243735634835-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 09:58:49.240202 13896 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 09:58:49.244021 13896 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 09:58:49.244267 13896 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 09:58:49.260910 13896 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:33167
I20260924 09:58:49.260941 13962 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:33167 every 8 connection(s)
I20260924 09:58:49.262177 13896 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790243735634835-388-0/minicluster-data/master-1/data/info.pb
I20260924 09:58:49.265748 13963 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 09:58:49.268151 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu as pid 13896
I20260924 09:58:49.268332 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790243735634835-388-0/minicluster-data/master-1/wal/instance
I20260924 09:58:49.268836 388 external_mini_cluster.cc:1428] Setting key 903cd3035d3d55c664253a7e74a9b72b
I20260924 09:58:49.273984 13963 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32781 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 09:58:49 dist-test-slave-w8d2 krb5kdc[13809](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243929, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.60@KRBTEST.COM" created.
I20260924 09:58:49.293315 13892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:58:49.277033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38576 (local address 127.0.97.62:32781)
0924 09:58:49.277208 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0924 09:58:49.277212 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 09:58:49.278460 (+ 1248us) server_negotiation.cc:407] Connection header received
0924 09:58:49.280267 (+ 1807us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:58:49.280272 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:58:49.280346 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:58:49.280431 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:58:49.281843 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:58:49.282517 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:58:49.284056 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:58:49.284468 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:58:49.284586 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:58:49.287957 (+ 3371us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:58:49.287961 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:58:49.287965 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:58:49.288004 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:58:49.290436 (+ 2432us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:58:49.291125 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:58:49.291128 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:58:49.291130 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:58:49.291283 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:58:49.291673 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:58:49.291678 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:58:49.291682 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:58:49.291964 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:58:49.292100 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0924 09:58:49.292785 (+ 685us) server_negotiation.cc:299] Negotiation successful
0924 09:58:49.293014 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 09:58:49.294183 13966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:58:49.277574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32781 (local address 127.0.0.1:38576)
0924 09:58:49.278308 (+ 734us) negotiation.cc:107] Waiting for socket to connect
0924 09:58:49.278352 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 09:58:49.280008 (+ 1656us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:58:49.280666 (+ 658us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:58:49.280674 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:58:49.281149 (+ 475us) client_negotiation.cc:189] Negotiated authn=SASL
0924 09:58:49.281682 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:58:49.281699 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:58:49.282659 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:58:49.282663 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:58:49.283923 (+ 1260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:58:49.283935 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:58:49.284669 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:58:49.284676 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:58:49.284878 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:58:49.285540 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 09:58:49.285561 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 09:58:49.287757 (+ 2196us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 09:58:49.290566 (+ 2809us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 09:58:49.290575 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 09:58:49.290591 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 09:58:49.291012 (+ 421us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 09:58:49.291378 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 09:58:49.291383 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 09:58:49.291387 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 09:58:49.291564 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 09:58:49.292089 (+ 525us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 09:58:49.292096 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 09:58:49.292413 (+ 317us) client_negotiation.cc:769] Sending connection context
0924 09:58:49.292657 (+ 244us) client_negotiation.cc:240] Negotiation successful
0924 09:58:49.293131 (+ 474us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":513,"thread_start_us":324,"threads_started":1}
I20260924 09:58:49.295895 13963 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33167 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 09:58:49 dist-test-slave-w8d2 krb5kdc[13809](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243929, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260924 09:58:49.309613 13966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:58:49.296614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:33167 (local address 127.0.0.1:59584)
0924 09:58:49.296801 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0924 09:58:49.296819 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 09:58:49.296896 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:58:49.297771 (+ 875us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:58:49.297776 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:58:49.298144 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0924 09:58:49.298268 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:58:49.298276 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:58:49.299565 (+ 1289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:58:49.299568 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:58:49.300369 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:58:49.300378 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:58:49.301103 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:58:49.301108 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:58:49.301272 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:58:49.301755 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 09:58:49.301770 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 09:58:49.303811 (+ 2041us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 09:58:49.307493 (+ 3682us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 09:58:49.307496 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 09:58:49.307499 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 09:58:49.307815 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 09:58:49.308190 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 09:58:49.308193 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 09:58:49.308195 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 09:58:49.308313 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 09:58:49.308835 (+ 522us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 09:58:49.308838 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 09:58:49.308990 (+ 152us) client_negotiation.cc:769] Sending connection context
0924 09:58:49.309113 (+ 123us) client_negotiation.cc:240] Negotiation successful
0924 09:58:49.309301 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260924 09:58:49.309824 13968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:58:49.296849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59584 (local address 127.0.97.61:33167)
0924 09:58:49.297402 (+ 553us) server_negotiation.cc:206] Beginning negotiation
0924 09:58:49.297408 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 09:58:49.297429 (+ 21us) server_negotiation.cc:407] Connection header received
0924 09:58:49.297467 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:58:49.297472 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:58:49.297641 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:58:49.297764 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:58:49.298406 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:58:49.299437 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:58:49.300482 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:58:49.300974 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:58:49.301075 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:58:49.304044 (+ 2969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:58:49.304059 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:58:49.304075 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:58:49.304112 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:58:49.307377 (+ 3265us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:58:49.307931 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:58:49.307940 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:58:49.307944 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:58:49.308097 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:58:49.308402 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:58:49.308405 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:58:49.308408 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:58:49.308735 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:58:49.308896 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0924 09:58:49.309249 (+ 353us) server_negotiation.cc:299] Negotiation successful
0924 09:58:49.309440 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":428,"thread_start_us":305,"threads_started":1}
I20260924 09:58:49.311944 13963 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46599 } has no permanent_uuid. Determining permanent_uuid...
I20260924 09:58:49.313026 13966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:58:49.312590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46599 (local address 127.0.0.1:55510)
0924 09:58:49.312763 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0924 09:58:49.312867 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260924 09:58:49.313274 13912 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46599: connect: Connection refused (error 111)
I20260924 09:58:49.314670 13966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:58:49.314372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46599 (local address 127.0.0.1:55522)
0924 09:58:49.314505 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0924 09:58:49.314582 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260924 09:58:49.314853 13963 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:46599: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46599: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790243735634835-388-0/krb5kdc/kudu_127.0.97.60.keytab.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790243735634835-388-0/krb5kdc/kudu_127.0.97.60.keytab.
I20260924 09:58:49.316562 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
/tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790243735634835-388-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790243735634835-388-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790243735634835-388-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790243735634835-388-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:46599
--webserver_interface=127.0.97.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790243735634835-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:32781,127.0.97.61:33167,127.0.97.60:46599
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790243735634835-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790243735634835-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790243735634835-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 09:58:49.364132 13963 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46599 } attempt: 1
I20260924 09:58:49.365235 13966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:58:49.364797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46599 (local address 127.0.0.1:55526)
0924 09:58:49.364980 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0924 09:58:49.365083 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260924 09:58:49.366182 13966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:58:49.365894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46599 (local address 127.0.0.1:55534)
0924 09:58:49.366011 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0924 09:58:49.366088 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260924 09:58:49.366371 13963 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:46599: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46599: connect: Connection refused (error 111)
I20260924 09:58:49.434589 13963 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46599 } attempt: 2
I20260924 09:58:49.435750 13966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:58:49.435306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46599 (local address 127.0.0.1:55544)
0924 09:58:49.435503 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0924 09:58:49.435595 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260924 09:58:49.436729 13966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:58:49.436404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46599 (local address 127.0.0.1:55554)
0924 09:58:49.436553 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0924 09:58:49.436621 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260924 09:58:49.436926 13963 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:46599: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46599: connect: Connection refused (error 111)
I20260924 09:58:49.509884 13889 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33167 } attempt: 5
Sep 24 09:58:49 dist-test-slave-w8d2 krb5kdc[13809](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243928, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260924 09:58:49.522921 13893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:58:49.510548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:33167 (local address 127.0.0.1:59594)
0924 09:58:49.510735 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0924 09:58:49.510756 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 09:58:49.510882 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:58:49.511261 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:58:49.511265 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:58:49.511682 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0924 09:58:49.511863 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:58:49.511872 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:58:49.512843 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:58:49.512847 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:58:49.513685 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:58:49.513696 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:58:49.514413 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:58:49.514419 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:58:49.514585 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:58:49.515277 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 09:58:49.515296 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 09:58:49.517520 (+ 2224us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 09:58:49.520471 (+ 2951us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 09:58:49.520476 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 09:58:49.520480 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 09:58:49.520973 (+ 493us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 09:58:49.521314 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 09:58:49.521319 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 09:58:49.521323 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 09:58:49.521486 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 09:58:49.521977 (+ 491us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 09:58:49.521982 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 09:58:49.522131 (+ 149us) client_negotiation.cc:769] Sending connection context
0924 09:58:49.522249 (+ 118us) client_negotiation.cc:240] Negotiation successful
0924 09:58:49.522580 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260924 09:58:49.522919 13968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:58:49.510683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59594 (local address 127.0.97.61:33167)
0924 09:58:49.510846 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0924 09:58:49.510851 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 09:58:49.510871 (+ 20us) server_negotiation.cc:407] Connection header received
0924 09:58:49.511017 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:58:49.511022 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:58:49.511097 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:58:49.511179 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:58:49.512024 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:58:49.512701 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:58:49.513822 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:58:49.514258 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:58:49.514369 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:58:49.517698 (+ 3329us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:58:49.517703 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:58:49.517707 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:58:49.517740 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:58:49.520323 (+ 2583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:58:49.521099 (+ 776us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:58:49.521104 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:58:49.521106 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:58:49.521223 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:58:49.521580 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:58:49.521583 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:58:49.521585 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:58:49.521850 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:58:49.521988 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0924 09:58:49.522357 (+ 369us) server_negotiation.cc:299] Negotiation successful
0924 09:58:49.522580 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 09:58:49.524190 13889 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46599 } has no permanent_uuid. Determining permanent_uuid...
I20260924 09:58:49.525256 13893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:58:49.524843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46599 (local address 127.0.0.1:55558)
0924 09:58:49.525033 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0924 09:58:49.525112 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260924 09:58:49.526118 13893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:58:49.525817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46599 (local address 127.0.0.1:55572)
0924 09:58:49.525950 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0924 09:58:49.526013 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260924 09:58:49.526299 13889 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:46599: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46599: connect: Connection refused (error 111)
I20260924 09:58:49.528103 13963 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46599 } attempt: 3
I20260924 09:58:49.528878 13966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:58:49.528536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46599 (local address 127.0.0.1:55588)
0924 09:58:49.528691 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0924 09:58:49.528759 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260924 09:58:49.529731 13966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:58:49.529360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46599 (local address 127.0.0.1:55590)
0924 09:58:49.529570 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0924 09:58:49.529632 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135}
W20260924 09:58:49.529912 13963 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:46599: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46599: connect: Connection refused (error 111)
I20260924 09:58:49.577518 13889 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46599 } attempt: 1
I20260924 09:58:49.578516 13893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:58:49.578089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46599 (local address 127.0.0.1:55592)
0924 09:58:49.578270 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0924 09:58:49.578355 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260924 09:58:49.579424 13893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:58:49.579096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46599 (local address 127.0.0.1:55598)
0924 09:58:49.579257 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0924 09:58:49.579327 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260924 09:58:49.579610 13889 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:46599: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46599: connect: Connection refused (error 111)
W20260924 09:58:49.623816 13970 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 09:58:49.624085 13970 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 09:58:49.624213 13970 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 09:58:49.631258 13970 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 09:58:49.631359 13970 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 09:58:49.631431 13970 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 09:58:49.631503 13970 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 09:58:49.631561 13970 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 09:58:49.640985 13970 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790243735634835-388-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790243735634835-388-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:32781,127.0.97.61:33167,127.0.97.60:46599
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790243735634835-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:46599
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790243735634835-388-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.97.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790243735634835-388-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 09:58:49.642194 13970 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 09:58:49.643458 13970 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 09:58:49.647835 13889 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46599 } attempt: 2
I20260924 09:58:49.648913 13893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:58:49.648448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46599 (local address 127.0.0.1:55602)
0924 09:58:49.648675 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0924 09:58:49.648764 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
I20260924 09:58:49.650063 13893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:58:49.649680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46599 (local address 127.0.0.1:55606)
0924 09:58:49.649835 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0924 09:58:49.649919 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260924 09:58:49.650270 13889 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:46599: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46599: connect: Connection refused (error 111)
W20260924 09:58:49.654433 13975 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:58:49.654500 13976 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:58:49.654522 13978 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 09:58:49.655094 13970 server_base.cc:1034] running on GCE node
I20260924 09:58:49.655804 13970 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 09:58:49.657063 13970 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 09:58:49.658254 13970 hybrid_clock.cc:648] HybridClock initialized: now 1790243929658212 us; error 31 us; skew 500 ppm
I20260924 09:58:49.658582 13970 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 09:58:49 dist-test-slave-w8d2 krb5kdc[13809](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243929, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 09:58:49.661350 13970 init.cc:377] Logged in from keytab as kudu/127.0.97.60@KRBTEST.COM (short username kudu)
I20260924 09:58:49.662846 13970 webserver.cc:460] Webserver started at http://127.0.97.60:41799/ using document root <none> and password file <none>
I20260924 09:58:49.663669 13970 fs_manager.cc:359] Metadata directory not provided
I20260924 09:58:49.663797 13970 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 09:58:49.664166 13970 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 09:58:49.667088 13970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790243735634835-388-0/minicluster-data/master-2/data/instance:
uuid: "81b9e5999ac24fc9b3db07d0c2743330"
format_stamp: "Formatted at 2026-09-24 09:58:49 on dist-test-slave-w8d2"
server_key: "9f189a8eaff0ecd6aa1e7c2babb60531"
server_key_iv: "8931f442d44824e22d722c5a9a937344"
server_key_version: "encryptionkey@0"
I20260924 09:58:49.667825 13970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790243735634835-388-0/minicluster-data/master-2/wal/instance:
uuid: "81b9e5999ac24fc9b3db07d0c2743330"
format_stamp: "Formatted at 2026-09-24 09:58:49 on dist-test-slave-w8d2"
server_key: "9f189a8eaff0ecd6aa1e7c2babb60531"
server_key_iv: "8931f442d44824e22d722c5a9a937344"
server_key_version: "encryptionkey@0"
I20260924 09:58:49.672448 13970 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 09:58:49.673085 13963 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46599 } attempt: 4
I20260924 09:58:49.674034 13966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:58:49.673637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46599 (local address 127.0.0.1:55610)
0924 09:58:49.673826 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0924 09:58:49.673898 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260924 09:58:49.675056 13966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:58:49.674693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46599 (local address 127.0.0.1:55616)
0924 09:58:49.674848 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0924 09:58:49.674929 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260924 09:58:49.675305 13963 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:46599: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46599: connect: Connection refused (error 111)
I20260924 09:58:49.676017 13985 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 09:58:49.677207 13970 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 09:58:49.677354 13970 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790243735634835-388-0/minicluster-data/master-2/data,/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790243735634835-388-0/minicluster-data/master-2/wal
uuid: "81b9e5999ac24fc9b3db07d0c2743330"
format_stamp: "Formatted at 2026-09-24 09:58:49 on dist-test-slave-w8d2"
server_key: "9f189a8eaff0ecd6aa1e7c2babb60531"
server_key_iv: "8931f442d44824e22d722c5a9a937344"
server_key_version: "encryptionkey@0"
I20260924 09:58:49.677527 13970 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790243735634835-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790243735634835-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790243735634835-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 09:58:49.690989 13970 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 09:58:49.694626 13970 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 09:58:49.694865 13970 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 09:58:49.710036 13970 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:46599
I20260924 09:58:49.710058 14036 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:46599 every 8 connection(s)
I20260924 09:58:49.711308 13970 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790243735634835-388-0/minicluster-data/master-2/data/info.pb
I20260924 09:58:49.714668 14037 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 09:58:49.715065 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu as pid 13970
I20260924 09:58:49.715210 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790243735634835-388-0/minicluster-data/master-2/wal/instance
I20260924 09:58:49.715631 388 external_mini_cluster.cc:1428] Setting key b532b0a485dac6fc80345601819c2f1b
I20260924 09:58:49.723492 14037 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32781 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 09:58:49 dist-test-slave-w8d2 krb5kdc[13809](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243929, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
Sep 24 09:58:49 dist-test-slave-w8d2 krb5kdc[13809](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243928, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260924 09:58:49.740196 13892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:58:49.726027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38584 (local address 127.0.97.62:32781)
0924 09:58:49.726224 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0924 09:58:49.726228 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 09:58:49.726786 (+ 558us) server_negotiation.cc:407] Connection header received
0924 09:58:49.726936 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:58:49.726942 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:58:49.727008 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:58:49.727085 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:58:49.728232 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:58:49.728882 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:58:49.730087 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:58:49.730503 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:58:49.730617 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:58:49.734464 (+ 3847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:58:49.734471 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:58:49.734475 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:58:49.734512 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:58:49.737106 (+ 2594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:58:49.737861 (+ 755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:58:49.737864 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:58:49.737866 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:58:49.737989 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:58:49.738417 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:58:49.738422 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:58:49.738426 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:58:49.738735 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:58:49.738895 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0924 09:58:49.739665 (+ 770us) server_negotiation.cc:299] Negotiation successful
0924 09:58:49.739832 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260924 09:58:49.741391 14040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:58:49.717467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55628 (local address 127.0.97.60:46599)
0924 09:58:49.718206 (+ 739us) server_negotiation.cc:206] Beginning negotiation
0924 09:58:49.718223 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0924 09:58:49.718269 (+ 46us) server_negotiation.cc:407] Connection header received
0924 09:58:49.720602 (+ 2333us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:58:49.720622 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:58:49.720994 (+ 372us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:58:49.721315 (+ 321us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:58:49.722969 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:58:49.724338 (+ 1369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:58:49.726107 (+ 1769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:58:49.727047 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:58:49.727212 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:58:49.735375 (+ 8163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:58:49.735383 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:58:49.735393 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:58:49.735419 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:58:49.738169 (+ 2750us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:58:49.738860 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:58:49.738868 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:58:49.738871 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:58:49.739010 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:58:49.739344 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:58:49.739349 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:58:49.739351 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:58:49.739744 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:58:49.739992 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0924 09:58:49.740464 (+ 472us) server_negotiation.cc:299] Negotiation successful
0924 09:58:49.740693 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":519,"thread_start_us":296,"threads_started":1}
I20260924 09:58:49.741422 14041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:58:49.725995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32781 (local address 127.0.0.1:38584)
0924 09:58:49.726669 (+ 674us) negotiation.cc:107] Waiting for socket to connect
0924 09:58:49.726702 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0924 09:58:49.726830 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:58:49.727162 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:58:49.727168 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:58:49.727793 (+ 625us) client_negotiation.cc:189] Negotiated authn=SASL
0924 09:58:49.728098 (+ 305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:58:49.728108 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:58:49.729012 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:58:49.729017 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:58:49.729964 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:58:49.729975 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:58:49.730707 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:58:49.730714 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:58:49.730856 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:58:49.732118 (+ 1262us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 09:58:49.732140 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 09:58:49.734247 (+ 2107us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 09:58:49.737222 (+ 2975us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 09:58:49.737236 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 09:58:49.737246 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0924 09:58:49.737771 (+ 525us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 09:58:49.738097 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 09:58:49.738102 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 09:58:49.738106 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 09:58:49.738320 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 09:58:49.738866 (+ 546us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 09:58:49.738873 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 09:58:49.739259 (+ 386us) client_negotiation.cc:769] Sending connection context
0924 09:58:49.739587 (+ 328us) client_negotiation.cc:240] Negotiation successful
0924 09:58:49.740013 (+ 426us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":370,"threads_started":1}
I20260924 09:58:49.742798 14037 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33167 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 09:58:49 dist-test-slave-w8d2 krb5kdc[13809](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243929, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
Sep 24 09:58:49 dist-test-slave-w8d2 krb5kdc[13809](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243928, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 09:58:49.756121 14041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:58:49.743374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:33167 (local address 127.0.0.1:59600)
0924 09:58:49.743502 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0924 09:58:49.743519 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 09:58:49.743609 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:58:49.743892 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:58:49.743895 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:58:49.744265 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0924 09:58:49.744427 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:58:49.744436 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:58:49.745370 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:58:49.745375 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:58:49.746203 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:58:49.746214 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:58:49.746864 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:58:49.746869 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:58:49.747056 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:58:49.747529 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 09:58:49.747543 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 09:58:49.750990 (+ 3447us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 09:58:49.753769 (+ 2779us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 09:58:49.753774 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 09:58:49.753778 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 09:58:49.754233 (+ 455us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 09:58:49.754543 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 09:58:49.754548 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 09:58:49.754552 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 09:58:49.754731 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 09:58:49.755278 (+ 547us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 09:58:49.755283 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 09:58:49.755430 (+ 147us) client_negotiation.cc:769] Sending connection context
0924 09:58:49.755575 (+ 145us) client_negotiation.cc:240] Negotiation successful
0924 09:58:49.755789 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260924 09:58:49.756126 13968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:58:49.743465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59600 (local address 127.0.97.61:33167)
0924 09:58:49.743627 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0924 09:58:49.743632 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 09:58:49.743651 (+ 19us) server_negotiation.cc:407] Connection header received
0924 09:58:49.743709 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:58:49.743714 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:58:49.743785 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:58:49.743877 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:58:49.744564 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:58:49.745240 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:58:49.746329 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:58:49.746745 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:58:49.746848 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:58:49.751185 (+ 4337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:58:49.751192 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:58:49.751197 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:58:49.751256 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:58:49.753653 (+ 2397us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:58:49.754329 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:58:49.754333 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:58:49.754335 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:58:49.754453 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:58:49.754829 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:58:49.754834 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:58:49.754837 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:58:49.755150 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:58:49.755313 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0924 09:58:49.755660 (+ 347us) server_negotiation.cc:299] Negotiation successful
0924 09:58:49.755874 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 09:58:49.756443 13889 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46599 } attempt: 3
I20260924 09:58:49.757514 13892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:58:49.744258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38600 (local address 127.0.97.62:32781)
0924 09:58:49.744399 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0924 09:58:49.744404 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 09:58:49.744459 (+ 55us) server_negotiation.cc:407] Connection header received
0924 09:58:49.744610 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:58:49.744615 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:58:49.744675 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:58:49.744747 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:58:49.745841 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:58:49.746610 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:58:49.747793 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:58:49.748181 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:58:49.748283 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:58:49.751983 (+ 3700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:58:49.751989 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:58:49.751993 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:58:49.752029 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:58:49.754646 (+ 2617us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:58:49.755375 (+ 729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:58:49.755380 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:58:49.755384 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:58:49.755519 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:58:49.755912 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:58:49.755917 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:58:49.755921 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:58:49.756232 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:58:49.756892 (+ 660us) server_negotiation.cc:1035] Waiting for connection context
0924 09:58:49.757024 (+ 132us) server_negotiation.cc:299] Negotiation successful
0924 09:58:49.757213 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 09:58:49.757583 14037 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46599 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 09:58:49 dist-test-slave-w8d2 krb5kdc[13809](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243928, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Sep 24 09:58:49 dist-test-slave-w8d2 krb5kdc[13809](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243929, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Sep 24 09:58:49 dist-test-slave-w8d2 krb5kdc[13809](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243928, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260924 09:58:49.770237 13893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:58:49.756936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46599 (local address 127.0.0.1:55642)
0924 09:58:49.757191 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0924 09:58:49.757208 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 09:58:49.757291 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:58:49.757576 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:58:49.757581 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:58:49.758000 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0924 09:58:49.758183 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:58:49.758199 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:58:49.759555 (+ 1356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:58:49.759560 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:58:49.760647 (+ 1087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:58:49.760660 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:58:49.761592 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:58:49.761598 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:58:49.761773 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:58:49.762278 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 09:58:49.762294 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 09:58:49.764360 (+ 2066us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 09:58:49.767779 (+ 3419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 09:58:49.767785 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 09:58:49.767789 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 09:58:49.768224 (+ 435us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 09:58:49.768551 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 09:58:49.768556 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 09:58:49.768560 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 09:58:49.768706 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 09:58:49.769315 (+ 609us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 09:58:49.769320 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 09:58:49.769504 (+ 184us) client_negotiation.cc:769] Sending connection context
0924 09:58:49.769641 (+ 137us) client_negotiation.cc:240] Negotiation successful
0924 09:58:49.769885 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124}
I20260924 09:58:49.771389 14040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:58:49.757087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55642 (local address 127.0.97.60:46599)
0924 09:58:49.757244 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0924 09:58:49.757249 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 09:58:49.757266 (+ 17us) server_negotiation.cc:407] Connection header received
0924 09:58:49.757389 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:58:49.757395 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:58:49.757457 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:58:49.757550 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:58:49.758318 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:58:49.759412 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:58:49.760815 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:58:49.761452 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:58:49.761576 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:58:49.764511 (+ 2935us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:58:49.764516 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:58:49.764521 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:58:49.764551 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:58:49.767657 (+ 3106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:58:49.768324 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:58:49.768328 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:58:49.768332 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:58:49.768444 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:58:49.768832 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:58:49.768836 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:58:49.768840 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:58:49.769175 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:58:49.770720 (+ 1545us) server_negotiation.cc:1035] Waiting for connection context
0924 09:58:49.770832 (+ 112us) server_negotiation.cc:299] Negotiation successful
0924 09:58:49.771043 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 09:58:49.773208 14041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:58:49.758269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46599 (local address 127.0.0.1:55650)
0924 09:58:49.759024 (+ 755us) negotiation.cc:107] Waiting for socket to connect
0924 09:58:49.759043 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 09:58:49.759122 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:58:49.759654 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:58:49.759657 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:58:49.760036 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0924 09:58:49.760201 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:58:49.760213 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:58:49.761695 (+ 1482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:58:49.761699 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:58:49.762979 (+ 1280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:58:49.762993 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:58:49.763870 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:58:49.763876 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:58:49.764083 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:58:49.765050 (+ 967us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 09:58:49.765066 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 09:58:49.767998 (+ 2932us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 09:58:49.770935 (+ 2937us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 09:58:49.770940 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 09:58:49.770944 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 09:58:49.771364 (+ 420us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 09:58:49.771700 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 09:58:49.771705 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 09:58:49.771709 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 09:58:49.771836 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 09:58:49.772296 (+ 460us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 09:58:49.772300 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 09:58:49.772427 (+ 127us) client_negotiation.cc:769] Sending connection context
0924 09:58:49.772560 (+ 133us) client_negotiation.cc:240] Negotiation successful
0924 09:58:49.772852 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":664}
I20260924 09:58:49.773408 14042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:58:49.758474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55650 (local address 127.0.97.60:46599)
0924 09:58:49.759196 (+ 722us) server_negotiation.cc:206] Beginning negotiation
0924 09:58:49.759201 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 09:58:49.759221 (+ 20us) server_negotiation.cc:407] Connection header received
0924 09:58:49.759285 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:58:49.759289 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:58:49.759355 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:58:49.759451 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:58:49.760374 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:58:49.761579 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:58:49.763123 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:58:49.763741 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:58:49.763857 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:58:49.768173 (+ 4316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:58:49.768177 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:58:49.768180 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:58:49.768204 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:58:49.770828 (+ 2624us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:58:49.771466 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:58:49.771470 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:58:49.771473 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:58:49.771604 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:58:49.771921 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:58:49.771926 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:58:49.771930 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:58:49.772193 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:58:49.772308 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0924 09:58:49.772665 (+ 357us) server_negotiation.cc:299] Negotiation successful
0924 09:58:49.772998 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":132,"server-negotiator.queue_time_us":595,"thread_start_us":375,"threads_started":1}
I20260924 09:58:49.774874 13968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:58:49.759351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59616 (local address 127.0.97.61:33167)
0924 09:58:49.759495 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0924 09:58:49.759499 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 09:58:49.759516 (+ 17us) server_negotiation.cc:407] Connection header received
0924 09:58:49.759567 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:58:49.759570 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:58:49.759631 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:58:49.759717 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:58:49.760966 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:58:49.761668 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:58:49.763321 (+ 1653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:58:49.763913 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:58:49.764016 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:58:49.768400 (+ 4384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:58:49.768406 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:58:49.768411 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:58:49.768441 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:58:49.771502 (+ 3061us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:58:49.772769 (+ 1267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:58:49.772774 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:58:49.772778 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:58:49.772939 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:58:49.773368 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:58:49.773373 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:58:49.773376 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:58:49.773686 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:58:49.773870 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0924 09:58:49.774300 (+ 430us) server_negotiation.cc:299] Negotiation successful
0924 09:58:49.774558 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 09:58:49.776716 13889 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c1be3585e7dc4a5e837562349eceb0f9: Bootstrap starting.
I20260924 09:58:49.778970 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 09:58:49.779875 13889 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c1be3585e7dc4a5e837562349eceb0f9: Neither blocks nor log segments found. Creating new log.
I20260924 09:58:49.780882 13889 log.cc:824] T 00000000000000000000000000000000 P c1be3585e7dc4a5e837562349eceb0f9: Log is configured to *not* fsync() on all Append() calls
I20260924 09:58:49.783402 13889 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c1be3585e7dc4a5e837562349eceb0f9: No bootstrap required, opened a new log
I20260924 09:58:49.784512 14037 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81b9e5999ac24fc9b3db07d0c2743330: Bootstrap starting.
I20260924 09:58:49.788383 14037 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 81b9e5999ac24fc9b3db07d0c2743330: Neither blocks nor log segments found. Creating new log.
I20260924 09:58:49.789606 14037 log.cc:824] T 00000000000000000000000000000000 P 81b9e5999ac24fc9b3db07d0c2743330: Log is configured to *not* fsync() on all Append() calls
I20260924 09:58:49.791796 13889 raft_consensus.cc:348] T 00000000000000000000000000000000 P c1be3585e7dc4a5e837562349eceb0f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1be3585e7dc4a5e837562349eceb0f9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32781 } } peers { permanent_uuid: "7eebe7d7bdfb4141aa7e6cbe632d79c5" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33167 } } peers { permanent_uuid: "81b9e5999ac24fc9b3db07d0c2743330" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46599 } }
I20260924 09:58:49.792254 13889 raft_consensus.cc:374] T 00000000000000000000000000000000 P c1be3585e7dc4a5e837562349eceb0f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 09:58:49.792377 13889 raft_consensus.cc:729] T 00000000000000000000000000000000 P c1be3585e7dc4a5e837562349eceb0f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1be3585e7dc4a5e837562349eceb0f9, State: Initialized, Role: FOLLOWER
I20260924 09:58:49.793038 13889 consensus_queue.cc:260] T 00000000000000000000000000000000 P c1be3585e7dc4a5e837562349eceb0f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1be3585e7dc4a5e837562349eceb0f9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32781 } } peers { permanent_uuid: "7eebe7d7bdfb4141aa7e6cbe632d79c5" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33167 } } peers { permanent_uuid: "81b9e5999ac24fc9b3db07d0c2743330" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46599 } }
I20260924 09:58:49.796957 14051 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1be3585e7dc4a5e837562349eceb0f9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1be3585e7dc4a5e837562349eceb0f9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32781 } } peers { permanent_uuid: "7eebe7d7bdfb4141aa7e6cbe632d79c5" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33167 } } peers { permanent_uuid: "81b9e5999ac24fc9b3db07d0c2743330" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46599 } } }
I20260924 09:58:49.800252 14037 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81b9e5999ac24fc9b3db07d0c2743330: No bootstrap required, opened a new log
I20260924 09:58:49.798617 13889 sys_catalog.cc:564] T 00000000000000000000000000000000 P c1be3585e7dc4a5e837562349eceb0f9 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 09:58:49.803373 14051 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1be3585e7dc4a5e837562349eceb0f9 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 09:58:49.810877 14037 raft_consensus.cc:348] T 00000000000000000000000000000000 P 81b9e5999ac24fc9b3db07d0c2743330 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1be3585e7dc4a5e837562349eceb0f9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32781 } } peers { permanent_uuid: "7eebe7d7bdfb4141aa7e6cbe632d79c5" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33167 } } peers { permanent_uuid: "81b9e5999ac24fc9b3db07d0c2743330" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46599 } }
I20260924 09:58:49.811306 14037 raft_consensus.cc:374] T 00000000000000000000000000000000 P 81b9e5999ac24fc9b3db07d0c2743330 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 09:58:49.811447 14037 raft_consensus.cc:729] T 00000000000000000000000000000000 P 81b9e5999ac24fc9b3db07d0c2743330 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81b9e5999ac24fc9b3db07d0c2743330, State: Initialized, Role: FOLLOWER
I20260924 09:58:49.812142 14037 consensus_queue.cc:260] T 00000000000000000000000000000000 P 81b9e5999ac24fc9b3db07d0c2743330 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1be3585e7dc4a5e837562349eceb0f9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32781 } } peers { permanent_uuid: "7eebe7d7bdfb4141aa7e6cbe632d79c5" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33167 } } peers { permanent_uuid: "81b9e5999ac24fc9b3db07d0c2743330" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46599 } }
I20260924 09:58:49.815490 13892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:58:49.790000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38608 (local address 127.0.97.62:32781)
0924 09:58:49.790166 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0924 09:58:49.790172 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 09:58:49.797614 (+ 7442us) server_negotiation.cc:407] Connection header received
0924 09:58:49.797877 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:58:49.797882 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:58:49.797957 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:58:49.798041 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:58:49.800322 (+ 2281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:58:49.801349 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:58:49.803423 (+ 2074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:58:49.804046 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:58:49.804143 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:58:49.807750 (+ 3607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:58:49.807757 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:58:49.807761 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:58:49.807797 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:58:49.811290 (+ 3493us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:58:49.812360 (+ 1070us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:58:49.812366 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:58:49.812371 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:58:49.812573 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:58:49.813029 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:58:49.813033 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:58:49.813037 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:58:49.813420 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:58:49.814695 (+ 1275us) server_negotiation.cc:1035] Waiting for connection context
0924 09:58:49.814836 (+ 141us) server_negotiation.cc:299] Negotiation successful
0924 09:58:49.815094 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 09:58:49.815985 14055 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81b9e5999ac24fc9b3db07d0c2743330 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1be3585e7dc4a5e837562349eceb0f9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32781 } } peers { permanent_uuid: "7eebe7d7bdfb4141aa7e6cbe632d79c5" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33167 } } peers { permanent_uuid: "81b9e5999ac24fc9b3db07d0c2743330" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46599 } } }
I20260924 09:58:49.817885 14055 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81b9e5999ac24fc9b3db07d0c2743330 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 09:58:49.817781 14037 sys_catalog.cc:564] T 00000000000000000000000000000000 P 81b9e5999ac24fc9b3db07d0c2743330 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 09:58:49.824884 13968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:58:49.790552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59618 (local address 127.0.97.61:33167)
0924 09:58:49.790719 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0924 09:58:49.790724 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 09:58:49.796925 (+ 6201us) server_negotiation.cc:407] Connection header received
0924 09:58:49.798931 (+ 2006us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:58:49.798937 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:58:49.799009 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:58:49.799076 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:58:49.801803 (+ 2727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:58:49.802866 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:58:49.807819 (+ 4953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:58:49.808415 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:58:49.808503 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:58:49.817857 (+ 9354us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:58:49.817865 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:58:49.817871 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:58:49.817915 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:58:49.821520 (+ 3605us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:58:49.822465 (+ 945us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:58:49.822471 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:58:49.822476 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:58:49.822648 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:58:49.823071 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:58:49.823077 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:58:49.823081 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:58:49.823465 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:58:49.823714 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0924 09:58:49.824249 (+ 535us) server_negotiation.cc:299] Negotiation successful
0924 09:58:49.824501 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 09:58:49.834820 14042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:58:49.796966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55658 (local address 127.0.97.60:46599)
0924 09:58:49.797143 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0924 09:58:49.797148 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 09:58:49.801618 (+ 4470us) server_negotiation.cc:407] Connection header received
0924 09:58:49.801749 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:58:49.801754 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:58:49.801825 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:58:49.801921 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:58:49.804017 (+ 2096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:58:49.805048 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:58:49.807072 (+ 2024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:58:49.807755 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:58:49.807850 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:58:49.826892 (+ 19042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:58:49.826901 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:58:49.826906 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:58:49.826950 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:58:49.830937 (+ 3987us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:58:49.832464 (+ 1527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:58:49.832471 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:58:49.832475 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:58:49.832630 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:58:49.833061 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:58:49.833066 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:58:49.833070 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:58:49.833445 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:58:49.833661 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0924 09:58:49.834222 (+ 561us) server_negotiation.cc:299] Negotiation successful
0924 09:58:49.834466 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
W20260924 09:58:49.843924 14066 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c1be3585e7dc4a5e837562349eceb0f9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 09:58:49.844125 14066 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 09:58:49.835150 13822 master_runner.cc:186] Error getting master registration for 127.0.97.61:33167: OK, instance_id { permanent_uuid: "7eebe7d7bdfb4141aa7e6cbe632d79c5" instance_seqno: 1790243929256182 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 09:58:49.848945 13822 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260924 09:58:49.856667 14076 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 81b9e5999ac24fc9b3db07d0c2743330: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 09:58:49.856906 14076 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 09:58:49.858664 13970 master_runner.cc:186] Error getting master registration for 127.0.97.61:33167: OK, instance_id { permanent_uuid: "7eebe7d7bdfb4141aa7e6cbe632d79c5" instance_seqno: 1790243929256182 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 09:58:49.866389 13970 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260924 09:58:49.946684 13991 tablet.cc:2369] T 00000000000000000000000000000000 P 81b9e5999ac24fc9b3db07d0c2743330: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 09:58:49.974511 13963 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46599 } attempt: 5
Sep 24 09:58:49 dist-test-slave-w8d2 krb5kdc[13809](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243929, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260924 09:58:49.987082 13966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:58:49.975111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46599 (local address 127.0.0.1:55674)
0924 09:58:49.975295 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0924 09:58:49.975315 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 09:58:49.975419 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:58:49.975759 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:58:49.975762 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:58:49.976209 (+ 447us) client_negotiation.cc:189] Negotiated authn=SASL
0924 09:58:49.976363 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:58:49.976372 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:58:49.977353 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:58:49.977359 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:58:49.978240 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:58:49.978251 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:58:49.978901 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:58:49.978907 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:58:49.979086 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:58:49.979647 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 09:58:49.979671 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 09:58:49.981842 (+ 2171us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 09:58:49.984844 (+ 3002us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 09:58:49.984850 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 09:58:49.984854 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 09:58:49.985214 (+ 360us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 09:58:49.985532 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 09:58:49.985537 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 09:58:49.985541 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 09:58:49.985675 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 09:58:49.986171 (+ 496us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 09:58:49.986175 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 09:58:49.986318 (+ 143us) client_negotiation.cc:769] Sending connection context
0924 09:58:49.986436 (+ 118us) client_negotiation.cc:240] Negotiation successful
0924 09:58:49.986701 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260924 09:58:49.987092 14042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:58:49.975210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55674 (local address 127.0.97.60:46599)
0924 09:58:49.975360 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0924 09:58:49.975365 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 09:58:49.975383 (+ 18us) server_negotiation.cc:407] Connection header received
0924 09:58:49.975542 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:58:49.975546 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:58:49.975617 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:58:49.975720 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:58:49.976496 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:58:49.977235 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:58:49.978369 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:58:49.978776 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:58:49.978876 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:58:49.982032 (+ 3156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:58:49.982038 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:58:49.982043 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:58:49.982077 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:58:49.984726 (+ 2649us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:58:49.985318 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:58:49.985322 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:58:49.985324 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:58:49.985435 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:58:49.985769 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:58:49.985774 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:58:49.985777 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:58:49.986059 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:58:49.986203 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0924 09:58:49.986555 (+ 352us) server_negotiation.cc:299] Negotiation successful
0924 09:58:49.986766 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 09:58:49.992362 13963 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7eebe7d7bdfb4141aa7e6cbe632d79c5: Bootstrap starting.
I20260924 09:58:49.996122 13963 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7eebe7d7bdfb4141aa7e6cbe632d79c5: Neither blocks nor log segments found. Creating new log.
I20260924 09:58:49.997241 13963 log.cc:824] T 00000000000000000000000000000000 P 7eebe7d7bdfb4141aa7e6cbe632d79c5: Log is configured to *not* fsync() on all Append() calls
I20260924 09:58:49.999650 13963 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7eebe7d7bdfb4141aa7e6cbe632d79c5: No bootstrap required, opened a new log
I20260924 09:58:50.006412 13963 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7eebe7d7bdfb4141aa7e6cbe632d79c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1be3585e7dc4a5e837562349eceb0f9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32781 } } peers { permanent_uuid: "7eebe7d7bdfb4141aa7e6cbe632d79c5" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33167 } } peers { permanent_uuid: "81b9e5999ac24fc9b3db07d0c2743330" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46599 } }
I20260924 09:58:50.006683 13963 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7eebe7d7bdfb4141aa7e6cbe632d79c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 09:58:50.006784 13963 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7eebe7d7bdfb4141aa7e6cbe632d79c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7eebe7d7bdfb4141aa7e6cbe632d79c5, State: Initialized, Role: FOLLOWER
I20260924 09:58:50.007325 13963 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7eebe7d7bdfb4141aa7e6cbe632d79c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1be3585e7dc4a5e837562349eceb0f9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32781 } } peers { permanent_uuid: "7eebe7d7bdfb4141aa7e6cbe632d79c5" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33167 } } peers { permanent_uuid: "81b9e5999ac24fc9b3db07d0c2743330" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46599 } }
I20260924 09:58:50.008127 14079 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7eebe7d7bdfb4141aa7e6cbe632d79c5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1be3585e7dc4a5e837562349eceb0f9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32781 } } peers { permanent_uuid: "7eebe7d7bdfb4141aa7e6cbe632d79c5" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33167 } } peers { permanent_uuid: "81b9e5999ac24fc9b3db07d0c2743330" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46599 } } }
I20260924 09:58:50.008553 14079 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7eebe7d7bdfb4141aa7e6cbe632d79c5 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 09:58:50.009207 13963 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7eebe7d7bdfb4141aa7e6cbe632d79c5 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 09:58:50.024539 13896 master_runner.cc:263] No leader master found from master 7eebe7d7bdfb4141aa7e6cbe632d79c5
I20260924 09:58:50.024682 13896 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260924 09:58:50.027143 14090 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7eebe7d7bdfb4141aa7e6cbe632d79c5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 09:58:50.027366 14090 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260924 09:58:50.045562 13843 tablet.cc:2369] T 00000000000000000000000000000000 P c1be3585e7dc4a5e837562349eceb0f9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 09:58:50.140731 14051 raft_consensus.cc:482] T 00000000000000000000000000000000 P c1be3585e7dc4a5e837562349eceb0f9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 09:58:50.141048 14051 raft_consensus.cc:504] T 00000000000000000000000000000000 P c1be3585e7dc4a5e837562349eceb0f9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1be3585e7dc4a5e837562349eceb0f9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32781 } } peers { permanent_uuid: "7eebe7d7bdfb4141aa7e6cbe632d79c5" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33167 } } peers { permanent_uuid: "81b9e5999ac24fc9b3db07d0c2743330" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46599 } }
I20260924 09:58:50.142554 14051 leader_election.cc:288] T 00000000000000000000000000000000 P c1be3585e7dc4a5e837562349eceb0f9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7eebe7d7bdfb4141aa7e6cbe632d79c5 (127.0.97.61:33167), 81b9e5999ac24fc9b3db07d0c2743330 (127.0.97.60:46599)
I20260924 09:58:50.143850 14012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c1be3585e7dc4a5e837562349eceb0f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "81b9e5999ac24fc9b3db07d0c2743330" is_pre_election: true
I20260924 09:58:50.144431 14012 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 81b9e5999ac24fc9b3db07d0c2743330 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c1be3585e7dc4a5e837562349eceb0f9 in term 0.
I20260924 09:58:50.145159 13838 leader_election.cc:302] T 00000000000000000000000000000000 P c1be3585e7dc4a5e837562349eceb0f9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 81b9e5999ac24fc9b3db07d0c2743330, c1be3585e7dc4a5e837562349eceb0f9; no voters:
I20260924 09:58:50.146132 13938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c1be3585e7dc4a5e837562349eceb0f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7eebe7d7bdfb4141aa7e6cbe632d79c5" is_pre_election: true
I20260924 09:58:50.146682 13938 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7eebe7d7bdfb4141aa7e6cbe632d79c5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c1be3585e7dc4a5e837562349eceb0f9 in term 0.
I20260924 09:58:50.147116 14051 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c1be3585e7dc4a5e837562349eceb0f9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 09:58:50.147267 14051 raft_consensus.cc:482] T 00000000000000000000000000000000 P c1be3585e7dc4a5e837562349eceb0f9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 09:58:50.147375 14051 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c1be3585e7dc4a5e837562349eceb0f9 [term 0 FOLLOWER]: Advancing to term 1
I20260924 09:58:50.150591 14051 raft_consensus.cc:504] T 00000000000000000000000000000000 P c1be3585e7dc4a5e837562349eceb0f9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1be3585e7dc4a5e837562349eceb0f9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32781 } } peers { permanent_uuid: "7eebe7d7bdfb4141aa7e6cbe632d79c5" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33167 } } peers { permanent_uuid: "81b9e5999ac24fc9b3db07d0c2743330" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46599 } }
I20260924 09:58:50.152096 13938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c1be3585e7dc4a5e837562349eceb0f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7eebe7d7bdfb4141aa7e6cbe632d79c5"
I20260924 09:58:50.152288 13938 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7eebe7d7bdfb4141aa7e6cbe632d79c5 [term 0 FOLLOWER]: Advancing to term 1
I20260924 09:58:50.158924 13938 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7eebe7d7bdfb4141aa7e6cbe632d79c5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c1be3585e7dc4a5e837562349eceb0f9 in term 1.
I20260924 09:58:50.159395 13840 leader_election.cc:302] T 00000000000000000000000000000000 P c1be3585e7dc4a5e837562349eceb0f9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7eebe7d7bdfb4141aa7e6cbe632d79c5, c1be3585e7dc4a5e837562349eceb0f9; no voters:
I20260924 09:58:50.164155 14012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c1be3585e7dc4a5e837562349eceb0f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "81b9e5999ac24fc9b3db07d0c2743330"
I20260924 09:58:50.164361 14012 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 81b9e5999ac24fc9b3db07d0c2743330 [term 0 FOLLOWER]: Advancing to term 1
I20260924 09:58:50.167732 14012 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 81b9e5999ac24fc9b3db07d0c2743330 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c1be3585e7dc4a5e837562349eceb0f9 in term 1.
I20260924 09:58:50.168419 14051 leader_election.cc:288] T 00000000000000000000000000000000 P c1be3585e7dc4a5e837562349eceb0f9 [CANDIDATE]: Term 1 election: Requested vote from peers 7eebe7d7bdfb4141aa7e6cbe632d79c5 (127.0.97.61:33167), 81b9e5999ac24fc9b3db07d0c2743330 (127.0.97.60:46599)
I20260924 09:58:50.168826 14051 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c1be3585e7dc4a5e837562349eceb0f9 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 09:58:50.169224 14051 raft_consensus.cc:686] T 00000000000000000000000000000000 P c1be3585e7dc4a5e837562349eceb0f9 [term 1 LEADER]: Becoming Leader. State: Replica: c1be3585e7dc4a5e837562349eceb0f9, State: Running, Role: LEADER
I20260924 09:58:50.169986 14051 consensus_queue.cc:237] T 00000000000000000000000000000000 P c1be3585e7dc4a5e837562349eceb0f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1be3585e7dc4a5e837562349eceb0f9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32781 } } peers { permanent_uuid: "7eebe7d7bdfb4141aa7e6cbe632d79c5" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33167 } } peers { permanent_uuid: "81b9e5999ac24fc9b3db07d0c2743330" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46599 } }
I20260924 09:58:50.186038 14051 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1be3585e7dc4a5e837562349eceb0f9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c1be3585e7dc4a5e837562349eceb0f9. Latest consensus state: current_term: 1 leader_uuid: "c1be3585e7dc4a5e837562349eceb0f9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1be3585e7dc4a5e837562349eceb0f9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32781 } } peers { permanent_uuid: "7eebe7d7bdfb4141aa7e6cbe632d79c5" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33167 } } peers { permanent_uuid: "81b9e5999ac24fc9b3db07d0c2743330" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46599 } } }
I20260924 09:58:50.186475 14051 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1be3585e7dc4a5e837562349eceb0f9 [sys.catalog]: This master's current role is: LEADER
I20260924 09:58:50.195749 14094 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 09:58:50.198127 14094 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 09:58:50.220150 13938 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7eebe7d7bdfb4141aa7e6cbe632d79c5 [term 1 FOLLOWER]: Refusing update from remote peer c1be3585e7dc4a5e837562349eceb0f9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 09:58:50.220157 14012 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 81b9e5999ac24fc9b3db07d0c2743330 [term 1 FOLLOWER]: Refusing update from remote peer c1be3585e7dc4a5e837562349eceb0f9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 09:58:50.221125 14092 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c1be3585e7dc4a5e837562349eceb0f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7eebe7d7bdfb4141aa7e6cbe632d79c5" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33167 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 09:58:50.221525 14051 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c1be3585e7dc4a5e837562349eceb0f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "81b9e5999ac24fc9b3db07d0c2743330" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 09:58:50.239727 14055 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81b9e5999ac24fc9b3db07d0c2743330 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c1be3585e7dc4a5e837562349eceb0f9. Latest consensus state: current_term: 1 leader_uuid: "c1be3585e7dc4a5e837562349eceb0f9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1be3585e7dc4a5e837562349eceb0f9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32781 } } peers { permanent_uuid: "7eebe7d7bdfb4141aa7e6cbe632d79c5" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33167 } } peers { permanent_uuid: "81b9e5999ac24fc9b3db07d0c2743330" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46599 } } }
I20260924 09:58:50.240037 14055 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81b9e5999ac24fc9b3db07d0c2743330 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 09:58:50.243556 14055 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81b9e5999ac24fc9b3db07d0c2743330 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c1be3585e7dc4a5e837562349eceb0f9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1be3585e7dc4a5e837562349eceb0f9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32781 } } peers { permanent_uuid: "7eebe7d7bdfb4141aa7e6cbe632d79c5" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33167 } } peers { permanent_uuid: "81b9e5999ac24fc9b3db07d0c2743330" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46599 } } }
I20260924 09:58:50.243820 14055 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81b9e5999ac24fc9b3db07d0c2743330 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 09:58:50.245798 14051 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1be3585e7dc4a5e837562349eceb0f9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c1be3585e7dc4a5e837562349eceb0f9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1be3585e7dc4a5e837562349eceb0f9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32781 } } peers { permanent_uuid: "7eebe7d7bdfb4141aa7e6cbe632d79c5" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33167 } } peers { permanent_uuid: "81b9e5999ac24fc9b3db07d0c2743330" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46599 } } }
I20260924 09:58:50.246129 14051 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1be3585e7dc4a5e837562349eceb0f9 [sys.catalog]: This master's current role is: LEADER
W20260924 09:58:50.246770 13917 tablet.cc:2369] T 00000000000000000000000000000000 P 7eebe7d7bdfb4141aa7e6cbe632d79c5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 09:58:50.247762 14097 mvcc.cc:205] Tried to move back new op lower bound from 7332839138170970112 to 7332839137988079616. Current Snapshot: MvccSnapshot[applied={T|T < 7332839138170970112}]
I20260924 09:58:50.247834 14094 catalog_manager.cc:1347] Generated new cluster ID: 1bbb08b77291472f932be704b9948ba1
I20260924 09:58:50.247927 14094 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 09:58:50.259305 14095 mvcc.cc:205] Tried to move back new op lower bound from 7332839138170970112 to 7332839137988079616. Current Snapshot: MvccSnapshot[applied={T|T < 7332839138170970112 or (T in {7332839138170970112})}]
I20260924 09:58:50.271669 14079 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7eebe7d7bdfb4141aa7e6cbe632d79c5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c1be3585e7dc4a5e837562349eceb0f9. Latest consensus state: current_term: 1 leader_uuid: "c1be3585e7dc4a5e837562349eceb0f9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1be3585e7dc4a5e837562349eceb0f9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32781 } } peers { permanent_uuid: "7eebe7d7bdfb4141aa7e6cbe632d79c5" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33167 } } peers { permanent_uuid: "81b9e5999ac24fc9b3db07d0c2743330" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46599 } } }
I20260924 09:58:50.271944 14079 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7eebe7d7bdfb4141aa7e6cbe632d79c5 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 09:58:50.273554 14092 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1be3585e7dc4a5e837562349eceb0f9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c1be3585e7dc4a5e837562349eceb0f9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1be3585e7dc4a5e837562349eceb0f9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32781 } } peers { permanent_uuid: "7eebe7d7bdfb4141aa7e6cbe632d79c5" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33167 } } peers { permanent_uuid: "81b9e5999ac24fc9b3db07d0c2743330" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46599 } } }
I20260924 09:58:50.273880 14092 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1be3585e7dc4a5e837562349eceb0f9 [sys.catalog]: This master's current role is: LEADER
I20260924 09:58:50.281318 14079 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7eebe7d7bdfb4141aa7e6cbe632d79c5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c1be3585e7dc4a5e837562349eceb0f9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1be3585e7dc4a5e837562349eceb0f9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32781 } } peers { permanent_uuid: "7eebe7d7bdfb4141aa7e6cbe632d79c5" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33167 } } peers { permanent_uuid: "81b9e5999ac24fc9b3db07d0c2743330" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46599 } } }
I20260924 09:58:50.281628 14079 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7eebe7d7bdfb4141aa7e6cbe632d79c5 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 09:58:50.308076 14094 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 09:58:50.310151 14094 catalog_manager.cc:1504] Loading token signing keys...
I20260924 09:58:50.327332 14094 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c1be3585e7dc4a5e837562349eceb0f9: Generated new TSK 0
I20260924 09:58:50.328534 14094 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 09:58:50.858700 14076 catalog_manager.cc:1259] Loaded cluster ID: 1bbb08b77291472f932be704b9948ba1
I20260924 09:58:50.858812 14076 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 81b9e5999ac24fc9b3db07d0c2743330: loading cluster ID for follower catalog manager: success
I20260924 09:58:50.861196 14076 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 81b9e5999ac24fc9b3db07d0c2743330: acquiring CA information for follower catalog manager: success
I20260924 09:58:50.862900 14076 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 81b9e5999ac24fc9b3db07d0c2743330: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 09:58:51.029347 14090 catalog_manager.cc:1259] Loaded cluster ID: 1bbb08b77291472f932be704b9948ba1
I20260924 09:58:51.029439 14090 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 7eebe7d7bdfb4141aa7e6cbe632d79c5: loading cluster ID for follower catalog manager: success
I20260924 09:58:51.031816 14090 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 7eebe7d7bdfb4141aa7e6cbe632d79c5: acquiring CA information for follower catalog manager: success
I20260924 09:58:51.033439 14090 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 7eebe7d7bdfb4141aa7e6cbe632d79c5: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 09:58:51.456735 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu pid 13822
I20260924 09:58:51.458024 13884 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 09:58:51.567219 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu with pid 13822
I20260924 09:58:51.583868 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu pid 13896
I20260924 09:58:51.584913 13958 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 09:58:51.688918 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu with pid 13896
I20260924 09:58:51.704924 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu pid 13970
I20260924 09:58:51.706008 14032 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 09:58:51.809996 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu with pid 13970
2026-09-24T09:58:51Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (4607 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790243735634835-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 09:58:51 dist-test-slave-w8d2 krb5kdc[14113](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 09:58:51 dist-test-slave-w8d2 krb5kdc[14113](info): set up 2 sockets
Sep 24 09:58:51 dist-test-slave-w8d2 krb5kdc[14113](info): commencing operation
krb5kdc: starting...
W20260924 09:58:52.815713 388 mini_kdc.cc:121] Time spent starting KDC: real 0.970s user 0.001s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790243735634835-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790243735634835-388-0/krb5kdc/test-user.keytab.
Sep 24 09:58:52 dist-test-slave-w8d2 krb5kdc[14113](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243932, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T09:58:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T09:58:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790243735634835-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790243735634835-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260924 09:58:53.015049 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
/tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790243735634835-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790243735634835-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790243735634835-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790243735634835-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33823
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790243735634835-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46217
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:33823
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790243735634835-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790243735634835-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790243735634835-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 09:58:53.320025 14126 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 09:58:53.320322 14126 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 09:58:53.320462 14126 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 09:58:53.327584 14126 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 09:58:53.327700 14126 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 09:58:53.327777 14126 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 09:58:53.327849 14126 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 09:58:53.327908 14126 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 09:58:53.337698 14126 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46217
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790243735634835-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790243735634835-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:33823
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790243735634835-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33823
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790243735634835-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790243735634835-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 09:58:53.339000 14126 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 09:58:53.340310 14126 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 09:58:53.351328 14131 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:58:53.351373 14132 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:58:53.351352 14134 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 09:58:53.352411 14126 server_base.cc:1034] running on GCE node
I20260924 09:58:53.353194 14126 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 09:58:53.354589 14126 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 09:58:53.355779 14126 hybrid_clock.cc:648] HybridClock initialized: now 1790243933355722 us; error 47 us; skew 500 ppm
I20260924 09:58:53.356132 14126 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 09:58:53 dist-test-slave-w8d2 krb5kdc[14113](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243933, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 09:58:53.358973 14126 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260924 09:58:53.360486 14126 webserver.cc:460] Webserver started at http://127.0.97.62:35357/ using document root <none> and password file <none>
I20260924 09:58:53.361171 14126 fs_manager.cc:359] Metadata directory not provided
I20260924 09:58:53.361287 14126 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 09:58:53.361570 14126 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 09:58:53.364544 14126 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790243735634835-388-0/minicluster-data/master-0/data/instance:
uuid: "6d8ff2c88a2c4b8ab5ca80e7b0f0ea56"
format_stamp: "Formatted at 2026-09-24 09:58:53 on dist-test-slave-w8d2"
server_key: "2433924f6e8916b978f63b6c8c0fbf63"
server_key_iv: "e5edbf5def2542673d749ec71824024d"
server_key_version: "encryptionkey@0"
I20260924 09:58:53.365266 14126 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790243735634835-388-0/minicluster-data/master-0/wal/instance:
uuid: "6d8ff2c88a2c4b8ab5ca80e7b0f0ea56"
format_stamp: "Formatted at 2026-09-24 09:58:53 on dist-test-slave-w8d2"
server_key: "2433924f6e8916b978f63b6c8c0fbf63"
server_key_iv: "e5edbf5def2542673d749ec71824024d"
server_key_version: "encryptionkey@0"
I20260924 09:58:53.370265 14126 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260924 09:58:53.373493 14141 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 09:58:53.374712 14126 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260924 09:58:53.374857 14126 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790243735634835-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790243735634835-388-0/minicluster-data/master-0/wal
uuid: "6d8ff2c88a2c4b8ab5ca80e7b0f0ea56"
format_stamp: "Formatted at 2026-09-24 09:58:53 on dist-test-slave-w8d2"
server_key: "2433924f6e8916b978f63b6c8c0fbf63"
server_key_iv: "e5edbf5def2542673d749ec71824024d"
server_key_version: "encryptionkey@0"
I20260924 09:58:53.375048 14126 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790243735634835-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790243735634835-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790243735634835-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 09:58:53.397567 14126 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 09:58:53.401749 14126 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 09:58:53.402010 14126 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 09:58:53.418203 14126 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:33823
I20260924 09:58:53.418221 14192 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:33823 every 8 connection(s)
I20260924 09:58:53.419628 14126 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790243735634835-388-0/minicluster-data/master-0/data/info.pb
I20260924 09:58:53.423209 14193 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 09:58:53.423374 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu as pid 14126
I20260924 09:58:53.423573 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790243735634835-388-0/minicluster-data/master-0/wal/instance
I20260924 09:58:53.424196 388 external_mini_cluster.cc:1428] Setting key 0e19b86544a33c9352dc1146a6259549
I20260924 09:58:53.436061 14193 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d8ff2c88a2c4b8ab5ca80e7b0f0ea56: Bootstrap starting.
Sep 24 09:58:53 dist-test-slave-w8d2 krb5kdc[14113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243932, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 09:58:53.439584 14193 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6d8ff2c88a2c4b8ab5ca80e7b0f0ea56: Neither blocks nor log segments found. Creating new log.
I20260924 09:58:53.440840 14193 log.cc:824] T 00000000000000000000000000000000 P 6d8ff2c88a2c4b8ab5ca80e7b0f0ea56: Log is configured to *not* fsync() on all Append() calls
I20260924 09:58:53.443877 14193 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d8ff2c88a2c4b8ab5ca80e7b0f0ea56: No bootstrap required, opened a new log
I20260924 09:58:53.446666 14196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:58:53.425871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39710 (local address 127.0.97.62:33823)
0924 09:58:53.426588 (+ 717us) server_negotiation.cc:206] Beginning negotiation
0924 09:58:53.426600 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 09:58:53.426639 (+ 39us) server_negotiation.cc:407] Connection header received
0924 09:58:53.428816 (+ 2177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:58:53.428839 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:58:53.429244 (+ 405us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:58:53.429567 (+ 323us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:58:53.431263 (+ 1696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:58:53.432533 (+ 1270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:58:53.434317 (+ 1784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:58:53.434976 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:58:53.435097 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:58:53.439537 (+ 4440us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:58:53.439558 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:58:53.439582 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:58:53.439630 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:58:53.442743 (+ 3113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:58:53.443543 (+ 800us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:58:53.443549 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:58:53.443555 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:58:53.443715 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:58:53.444114 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:58:53.444120 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:58:53.444122 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:58:53.444663 (+ 541us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:58:53.444899 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0924 09:58:53.445289 (+ 390us) server_negotiation.cc:299] Negotiation successful
0924 09:58:53.445642 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":523,"thread_start_us":322,"threads_started":1}
I20260924 09:58:53.450250 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 09:58:53.452253 14193 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6d8ff2c88a2c4b8ab5ca80e7b0f0ea56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d8ff2c88a2c4b8ab5ca80e7b0f0ea56" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33823 } }
I20260924 09:58:53.452653 14193 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6d8ff2c88a2c4b8ab5ca80e7b0f0ea56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 09:58:53.452759 14193 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6d8ff2c88a2c4b8ab5ca80e7b0f0ea56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d8ff2c88a2c4b8ab5ca80e7b0f0ea56, State: Initialized, Role: FOLLOWER
I20260924 09:58:53.453347 14193 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6d8ff2c88a2c4b8ab5ca80e7b0f0ea56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d8ff2c88a2c4b8ab5ca80e7b0f0ea56" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33823 } }
I20260924 09:58:53.453567 14193 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6d8ff2c88a2c4b8ab5ca80e7b0f0ea56 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 09:58:53.453696 14193 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6d8ff2c88a2c4b8ab5ca80e7b0f0ea56 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 09:58:53.453848 14193 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6d8ff2c88a2c4b8ab5ca80e7b0f0ea56 [term 0 FOLLOWER]: Advancing to term 1
I20260924 09:58:53.455955 14193 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6d8ff2c88a2c4b8ab5ca80e7b0f0ea56 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d8ff2c88a2c4b8ab5ca80e7b0f0ea56" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33823 } }
I20260924 09:58:53.456396 14193 leader_election.cc:302] T 00000000000000000000000000000000 P 6d8ff2c88a2c4b8ab5ca80e7b0f0ea56 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6d8ff2c88a2c4b8ab5ca80e7b0f0ea56; no voters:
I20260924 09:58:53.456882 14193 leader_election.cc:288] T 00000000000000000000000000000000 P 6d8ff2c88a2c4b8ab5ca80e7b0f0ea56 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 09:58:53.457010 14198 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6d8ff2c88a2c4b8ab5ca80e7b0f0ea56 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 09:58:53.457587 14198 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6d8ff2c88a2c4b8ab5ca80e7b0f0ea56 [term 1 LEADER]: Becoming Leader. State: Replica: 6d8ff2c88a2c4b8ab5ca80e7b0f0ea56, State: Running, Role: LEADER
I20260924 09:58:53.458071 14198 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6d8ff2c88a2c4b8ab5ca80e7b0f0ea56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d8ff2c88a2c4b8ab5ca80e7b0f0ea56" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33823 } }
I20260924 09:58:53.458714 14193 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6d8ff2c88a2c4b8ab5ca80e7b0f0ea56 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 09:58:53.460935 14200 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d8ff2c88a2c4b8ab5ca80e7b0f0ea56 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6d8ff2c88a2c4b8ab5ca80e7b0f0ea56. Latest consensus state: current_term: 1 leader_uuid: "6d8ff2c88a2c4b8ab5ca80e7b0f0ea56" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d8ff2c88a2c4b8ab5ca80e7b0f0ea56" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33823 } } }
I20260924 09:58:53.461402 14200 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d8ff2c88a2c4b8ab5ca80e7b0f0ea56 [sys.catalog]: This master's current role is: LEADER
I20260924 09:58:53.460945 14199 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d8ff2c88a2c4b8ab5ca80e7b0f0ea56 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6d8ff2c88a2c4b8ab5ca80e7b0f0ea56" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d8ff2c88a2c4b8ab5ca80e7b0f0ea56" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33823 } } }
I20260924 09:58:53.461894 14199 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d8ff2c88a2c4b8ab5ca80e7b0f0ea56 [sys.catalog]: This master's current role is: LEADER
I20260924 09:58:53.462697 14206 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 09:58:53.466900 14206 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 09:58:53.474201 14206 catalog_manager.cc:1347] Generated new cluster ID: 181f369440184e6b88f6333bf9a6269a
I20260924 09:58:53.474300 14206 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 09:58:53.504479 14206 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 09:58:53.505972 14206 catalog_manager.cc:1504] Loading token signing keys...
I20260924 09:58:53.511742 14206 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6d8ff2c88a2c4b8ab5ca80e7b0f0ea56: Generated new TSK 0
I20260924 09:58:53.512751 14206 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 09:58:53.563009 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu pid 14126
I20260924 09:58:53.564242 14188 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 09:58:53.659360 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu with pid 14126
2026-09-24T09:58:53Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1846 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 09:58:53 dist-test-slave-w8d2 krb5kdc[14221](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 09:58:53 dist-test-slave-w8d2 krb5kdc[14221](info): set up 2 sockets
Sep 24 09:58:53 dist-test-slave-w8d2 krb5kdc[14221](info): commencing operation
krb5kdc: starting...
W20260924 09:58:54.655465 388 mini_kdc.cc:121] Time spent starting KDC: real 0.963s user 0.006s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/krb5kdc/test-user.keytab.
Sep 24 09:58:54 dist-test-slave-w8d2 krb5kdc[14221](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243934, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T09:58:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T09:58:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260924 09:58:54.868810 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
/tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46551
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:46551
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 09:58:55.175202 14234 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 09:58:55.175529 14234 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 09:58:55.175664 14234 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 09:58:55.182724 14234 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 09:58:55.182839 14234 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 09:58:55.182919 14234 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 09:58:55.182989 14234 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 09:58:55.183074 14234 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 09:58:55.192754 14234 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:46551
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46551
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 09:58:55.194010 14234 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 09:58:55.195286 14234 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 09:58:55.206385 14242 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:58:55.206459 14239 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:58:55.206429 14240 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 09:58:55.207073 14234 server_base.cc:1034] running on GCE node
I20260924 09:58:55.207836 14234 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 09:58:55.209112 14234 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 09:58:55.210304 14234 hybrid_clock.cc:648] HybridClock initialized: now 1790243935210267 us; error 26 us; skew 500 ppm
I20260924 09:58:55.210615 14234 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 09:58:55 dist-test-slave-w8d2 krb5kdc[14221](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243935, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 09:58:55.216724 14234 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260924 09:58:55.218151 14234 webserver.cc:460] Webserver started at http://127.0.97.62:38575/ using document root <none> and password file <none>
I20260924 09:58:55.218806 14234 fs_manager.cc:359] Metadata directory not provided
I20260924 09:58:55.218909 14234 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 09:58:55.219192 14234 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 09:58:55.222070 14234 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/minicluster-data/master-0/data/instance:
uuid: "76cdda25f78f4f8dab4b5e8cdf63a2b7"
format_stamp: "Formatted at 2026-09-24 09:58:55 on dist-test-slave-w8d2"
server_key: "455e524ee257bc375ab1961387ac0eca"
server_key_iv: "8c564b7cccd5eb633989057664a5c375"
server_key_version: "encryptionkey@0"
I20260924 09:58:55.222760 14234 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/minicluster-data/master-0/wal/instance:
uuid: "76cdda25f78f4f8dab4b5e8cdf63a2b7"
format_stamp: "Formatted at 2026-09-24 09:58:55 on dist-test-slave-w8d2"
server_key: "455e524ee257bc375ab1961387ac0eca"
server_key_iv: "8c564b7cccd5eb633989057664a5c375"
server_key_version: "encryptionkey@0"
I20260924 09:58:55.227682 14234 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260924 09:58:55.230815 14249 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 09:58:55.232074 14234 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 09:58:55.232226 14234 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/minicluster-data/master-0/wal
uuid: "76cdda25f78f4f8dab4b5e8cdf63a2b7"
format_stamp: "Formatted at 2026-09-24 09:58:55 on dist-test-slave-w8d2"
server_key: "455e524ee257bc375ab1961387ac0eca"
server_key_iv: "8c564b7cccd5eb633989057664a5c375"
server_key_version: "encryptionkey@0"
I20260924 09:58:55.232407 14234 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 09:58:55.244908 14234 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 09:58:55.249089 14234 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 09:58:55.249348 14234 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 09:58:55.265833 14234 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:46551
I20260924 09:58:55.265852 14300 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:46551 every 8 connection(s)
I20260924 09:58:55.267266 14234 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/minicluster-data/master-0/data/info.pb
I20260924 09:58:55.270929 14301 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 09:58:55.277243 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu as pid 14234
I20260924 09:58:55.277446 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/minicluster-data/master-0/wal/instance
I20260924 09:58:55.278049 388 external_mini_cluster.cc:1428] Setting key 6f747864c87d961d709bbc39ad8624e0
I20260924 09:58:55.281562 14301 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76cdda25f78f4f8dab4b5e8cdf63a2b7: Bootstrap starting.
I20260924 09:58:55.285286 14301 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 76cdda25f78f4f8dab4b5e8cdf63a2b7: Neither blocks nor log segments found. Creating new log.
I20260924 09:58:55.286554 14301 log.cc:824] T 00000000000000000000000000000000 P 76cdda25f78f4f8dab4b5e8cdf63a2b7: Log is configured to *not* fsync() on all Append() calls
I20260924 09:58:55.289146 14301 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76cdda25f78f4f8dab4b5e8cdf63a2b7: No bootstrap required, opened a new log
Sep 24 09:58:55 dist-test-slave-w8d2 krb5kdc[14221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243934, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 09:58:55.296602 14301 raft_consensus.cc:348] T 00000000000000000000000000000000 P 76cdda25f78f4f8dab4b5e8cdf63a2b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76cdda25f78f4f8dab4b5e8cdf63a2b7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46551 } }
I20260924 09:58:55.296931 14301 raft_consensus.cc:374] T 00000000000000000000000000000000 P 76cdda25f78f4f8dab4b5e8cdf63a2b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 09:58:55.297031 14301 raft_consensus.cc:729] T 00000000000000000000000000000000 P 76cdda25f78f4f8dab4b5e8cdf63a2b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76cdda25f78f4f8dab4b5e8cdf63a2b7, State: Initialized, Role: FOLLOWER
I20260924 09:58:55.297612 14301 consensus_queue.cc:260] T 00000000000000000000000000000000 P 76cdda25f78f4f8dab4b5e8cdf63a2b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76cdda25f78f4f8dab4b5e8cdf63a2b7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46551 } }
I20260924 09:58:55.297835 14301 raft_consensus.cc:388] T 00000000000000000000000000000000 P 76cdda25f78f4f8dab4b5e8cdf63a2b7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 09:58:55.297967 14301 raft_consensus.cc:482] T 00000000000000000000000000000000 P 76cdda25f78f4f8dab4b5e8cdf63a2b7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 09:58:55.298122 14301 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 76cdda25f78f4f8dab4b5e8cdf63a2b7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 09:58:55.300349 14304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:58:55.279999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47724 (local address 127.0.97.62:46551)
0924 09:58:55.280726 (+ 727us) server_negotiation.cc:206] Beginning negotiation
0924 09:58:55.280742 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 09:58:55.280794 (+ 52us) server_negotiation.cc:407] Connection header received
0924 09:58:55.283100 (+ 2306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:58:55.283134 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:58:55.283693 (+ 559us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:58:55.284130 (+ 437us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:58:55.285312 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:58:55.287224 (+ 1912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:58:55.288501 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:58:55.289151 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:58:55.289323 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:58:55.293168 (+ 3845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:58:55.293175 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:58:55.293189 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:58:55.293224 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:58:55.296355 (+ 3131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:58:55.296990 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:58:55.296997 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:58:55.297003 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:58:55.297209 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:58:55.297568 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:58:55.297573 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:58:55.297576 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:58:55.298203 (+ 627us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:58:55.298484 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0924 09:58:55.298942 (+ 458us) server_negotiation.cc:299] Negotiation successful
0924 09:58:55.299402 (+ 460us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":503,"thread_start_us":305,"threads_started":1}
I20260924 09:58:55.300388 14301 raft_consensus.cc:504] T 00000000000000000000000000000000 P 76cdda25f78f4f8dab4b5e8cdf63a2b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76cdda25f78f4f8dab4b5e8cdf63a2b7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46551 } }
I20260924 09:58:55.300879 14301 leader_election.cc:302] T 00000000000000000000000000000000 P 76cdda25f78f4f8dab4b5e8cdf63a2b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 76cdda25f78f4f8dab4b5e8cdf63a2b7; no voters:
I20260924 09:58:55.301519 14301 leader_election.cc:288] T 00000000000000000000000000000000 P 76cdda25f78f4f8dab4b5e8cdf63a2b7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 09:58:55.301688 14306 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 76cdda25f78f4f8dab4b5e8cdf63a2b7 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 09:58:55.303591 14306 raft_consensus.cc:686] T 00000000000000000000000000000000 P 76cdda25f78f4f8dab4b5e8cdf63a2b7 [term 1 LEADER]: Becoming Leader. State: Replica: 76cdda25f78f4f8dab4b5e8cdf63a2b7, State: Running, Role: LEADER
I20260924 09:58:55.303659 14301 sys_catalog.cc:564] T 00000000000000000000000000000000 P 76cdda25f78f4f8dab4b5e8cdf63a2b7 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 09:58:55.304128 14306 consensus_queue.cc:237] T 00000000000000000000000000000000 P 76cdda25f78f4f8dab4b5e8cdf63a2b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76cdda25f78f4f8dab4b5e8cdf63a2b7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46551 } }
I20260924 09:58:55.306741 14308 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76cdda25f78f4f8dab4b5e8cdf63a2b7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 76cdda25f78f4f8dab4b5e8cdf63a2b7. Latest consensus state: current_term: 1 leader_uuid: "76cdda25f78f4f8dab4b5e8cdf63a2b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76cdda25f78f4f8dab4b5e8cdf63a2b7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46551 } } }
I20260924 09:58:55.307088 14308 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76cdda25f78f4f8dab4b5e8cdf63a2b7 [sys.catalog]: This master's current role is: LEADER
I20260924 09:58:55.306726 14307 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76cdda25f78f4f8dab4b5e8cdf63a2b7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "76cdda25f78f4f8dab4b5e8cdf63a2b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76cdda25f78f4f8dab4b5e8cdf63a2b7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46551 } } }
I20260924 09:58:55.307534 14307 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76cdda25f78f4f8dab4b5e8cdf63a2b7 [sys.catalog]: This master's current role is: LEADER
I20260924 09:58:55.308820 14313 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 09:58:55.313022 14313 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 09:58:55.321215 14313 catalog_manager.cc:1347] Generated new cluster ID: e58d63fbf0b544da81d1e5cd9e6c8806
I20260924 09:58:55.321326 14313 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 09:58:55.335793 14313 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 09:58:55.337200 14313 catalog_manager.cc:1504] Loading token signing keys...
I20260924 09:58:55.349829 14313 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 76cdda25f78f4f8dab4b5e8cdf63a2b7: Generated new TSK 0
I20260924 09:58:55.350684 14313 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260924 09:58:55.402491 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
/tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46551
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 09:58:55.716482 14327 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 09:58:55.716755 14327 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 09:58:55.716885 14327 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 09:58:55.724093 14327 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 09:58:55.724220 14327 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 09:58:55.724377 14327 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260924 09:58:55.734175 14327 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46551
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 09:58:55.735528 14327 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 09:58:55.736850 14327 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 09:58:55.752365 14333 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:58:55.752532 14332 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:58:55.752908 14335 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 09:58:55.753681 14327 server_base.cc:1034] running on GCE node
I20260924 09:58:55.754412 14327 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 09:58:55.755618 14327 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 09:58:55.756827 14327 hybrid_clock.cc:648] HybridClock initialized: now 1790243935756701 us; error 114 us; skew 500 ppm
I20260924 09:58:55.757189 14327 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 09:58:55 dist-test-slave-w8d2 krb5kdc[14221](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243935, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 09:58:55.761067 14327 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260924 09:58:55.764447 14327 webserver.cc:460] Webserver started at http://127.0.97.1:44965/ using document root <none> and password file <none>
I20260924 09:58:55.765285 14327 fs_manager.cc:359] Metadata directory not provided
I20260924 09:58:55.765404 14327 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 09:58:55.765694 14327 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 09:58:55.768571 14327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/minicluster-data/ts-0/data/instance:
uuid: "fc45f069420445869c886224da489dc6"
format_stamp: "Formatted at 2026-09-24 09:58:55 on dist-test-slave-w8d2"
server_key: "49da492ccbc870cadd378684801b7529"
server_key_iv: "82c3f8a4747d352eaf96db4cf43ace10"
server_key_version: "encryptionkey@0"
I20260924 09:58:55.769299 14327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/minicluster-data/ts-0/wal/instance:
uuid: "fc45f069420445869c886224da489dc6"
format_stamp: "Formatted at 2026-09-24 09:58:55 on dist-test-slave-w8d2"
server_key: "49da492ccbc870cadd378684801b7529"
server_key_iv: "82c3f8a4747d352eaf96db4cf43ace10"
server_key_version: "encryptionkey@0"
I20260924 09:58:55.774112 14327 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260924 09:58:55.777379 14342 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 09:58:55.778558 14327 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260924 09:58:55.778699 14327 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/minicluster-data/ts-0/wal
uuid: "fc45f069420445869c886224da489dc6"
format_stamp: "Formatted at 2026-09-24 09:58:55 on dist-test-slave-w8d2"
server_key: "49da492ccbc870cadd378684801b7529"
server_key_iv: "82c3f8a4747d352eaf96db4cf43ace10"
server_key_version: "encryptionkey@0"
I20260924 09:58:55.778877 14327 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 09:58:55.812740 14327 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 09:58:55.816952 14327 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 09:58:55.817206 14327 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 09:58:55.818135 14327 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 09:58:55.819448 14327 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 09:58:55.819526 14327 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 09:58:55.819644 14327 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 09:58:55.819694 14327 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 09:58:55.848495 14327 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:36377
I20260924 09:58:55.848526 14454 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:36377 every 8 connection(s)
I20260924 09:58:55.850124 14327 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/minicluster-data/ts-0/data/info.pb
I20260924 09:58:55.852712 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu as pid 14327
I20260924 09:58:55.852914 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/minicluster-data/ts-0/wal/instance
I20260924 09:58:55.853678 388 external_mini_cluster.cc:1428] Setting key 63f06306e1e25ae0f71dacaeaa315f03
Sep 24 09:58:55 dist-test-slave-w8d2 krb5kdc[14221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243935, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 09:58:55.870828 14457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:58:55.852985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:46605 (local address 127.0.97.62:46551)
0924 09:58:55.853644 (+ 659us) server_negotiation.cc:206] Beginning negotiation
0924 09:58:55.853649 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 09:58:55.854552 (+ 903us) server_negotiation.cc:407] Connection header received
0924 09:58:55.856777 (+ 2225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:58:55.856781 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:58:55.856843 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:58:55.856924 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:58:55.858807 (+ 1883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:58:55.859545 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:58:55.861631 (+ 2086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:58:55.862011 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:58:55.862099 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:58:55.865615 (+ 3516us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:58:55.865620 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:58:55.865623 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:58:55.865660 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:58:55.868026 (+ 2366us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:58:55.868818 (+ 792us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:58:55.868822 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:58:55.868824 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:58:55.868928 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:58:55.869317 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:58:55.869320 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:58:55.869321 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:58:55.869556 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:58:55.869660 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0924 09:58:55.870407 (+ 747us) server_negotiation.cc:299] Negotiation successful
0924 09:58:55.870563 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":535,"thread_start_us":435,"threads_started":1}
I20260924 09:58:55.872138 14458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:58:55.853615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46551 (local address 127.0.97.1:46605)
0924 09:58:55.854376 (+ 761us) negotiation.cc:107] Waiting for socket to connect
0924 09:58:55.854414 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 09:58:55.856486 (+ 2072us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:58:55.857195 (+ 709us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:58:55.857205 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:58:55.857876 (+ 671us) client_negotiation.cc:189] Negotiated authn=SASL
0924 09:58:55.858656 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:58:55.858678 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:58:55.859669 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:58:55.859674 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:58:55.861425 (+ 1751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:58:55.861438 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:58:55.862207 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:58:55.862214 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:58:55.862461 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:58:55.863100 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 09:58:55.863143 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0924 09:58:55.865397 (+ 2254us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 09:58:55.868139 (+ 2742us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 09:58:55.868148 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 09:58:55.868164 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 09:58:55.868728 (+ 564us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 09:58:55.869009 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 09:58:55.869014 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 09:58:55.869017 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 09:58:55.869229 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 09:58:55.869654 (+ 425us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 09:58:55.869661 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 09:58:55.870033 (+ 372us) client_negotiation.cc:769] Sending connection context
0924 09:58:55.870327 (+ 294us) client_negotiation.cc:240] Negotiation successful
0924 09:58:55.870796 (+ 469us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":281,"threads_started":1}
I20260924 09:58:55.873590 14455 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46551
I20260924 09:58:55.873855 14455 heartbeater.cc:464] Registering TS with master...
I20260924 09:58:55.874528 14455 heartbeater.cc:511] Master 127.0.97.62:46551 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260924 09:58:55.876461 14266 ts_manager.cc:194] Registered new tserver with Master: fc45f069420445869c886224da489dc6 (127.0.97.1:36377)
I20260924 09:58:55.877961 14266 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:46605
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260924 09:58:55.902020 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
/tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46551
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 09:58:56.205282 14461 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 09:58:56.205547 14461 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 09:58:56.205648 14461 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 09:58:56.213207 14461 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 09:58:56.213310 14461 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 09:58:56.213426 14461 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260924 09:58:56.222888 14461 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46551
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 09:58:56.224112 14461 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 09:58:56.225345 14461 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 09:58:56.237726 14467 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:58:56.237783 14466 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:58:56.237754 14469 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 09:58:56.238391 14461 server_base.cc:1034] running on GCE node
I20260924 09:58:56.238924 14461 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 09:58:56.239912 14461 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 09:58:56.241113 14461 hybrid_clock.cc:648] HybridClock initialized: now 1790243936241053 us; error 51 us; skew 500 ppm
I20260924 09:58:56.241425 14461 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 09:58:56 dist-test-slave-w8d2 krb5kdc[14221](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243936, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 09:58:56.244438 14461 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260924 09:58:56.245877 14461 webserver.cc:460] Webserver started at http://127.0.97.2:44365/ using document root <none> and password file <none>
I20260924 09:58:56.246536 14461 fs_manager.cc:359] Metadata directory not provided
I20260924 09:58:56.246642 14461 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 09:58:56.246927 14461 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 09:58:56.249812 14461 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/minicluster-data/ts-1/data/instance:
uuid: "d540747f7bab4b0aa074cda755e1642d"
format_stamp: "Formatted at 2026-09-24 09:58:56 on dist-test-slave-w8d2"
server_key: "0fa54e7f8d9fcddc5e832038d95a6d98"
server_key_iv: "3bb8091213b5eacbce3be064ffdb49ec"
server_key_version: "encryptionkey@0"
I20260924 09:58:56.250558 14461 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/minicluster-data/ts-1/wal/instance:
uuid: "d540747f7bab4b0aa074cda755e1642d"
format_stamp: "Formatted at 2026-09-24 09:58:56 on dist-test-slave-w8d2"
server_key: "0fa54e7f8d9fcddc5e832038d95a6d98"
server_key_iv: "3bb8091213b5eacbce3be064ffdb49ec"
server_key_version: "encryptionkey@0"
I20260924 09:58:56.255728 14461 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260924 09:58:56.258880 14476 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 09:58:56.260130 14461 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 09:58:56.260293 14461 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/minicluster-data/ts-1/wal
uuid: "d540747f7bab4b0aa074cda755e1642d"
format_stamp: "Formatted at 2026-09-24 09:58:56 on dist-test-slave-w8d2"
server_key: "0fa54e7f8d9fcddc5e832038d95a6d98"
server_key_iv: "3bb8091213b5eacbce3be064ffdb49ec"
server_key_version: "encryptionkey@0"
I20260924 09:58:56.260465 14461 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 09:58:56.283596 14461 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 09:58:56.287498 14461 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 09:58:56.287751 14461 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 09:58:56.288712 14461 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 09:58:56.290019 14461 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 09:58:56.290093 14461 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 09:58:56.290213 14461 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 09:58:56.290261 14461 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 09:58:56.319507 14461 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:40223
I20260924 09:58:56.319554 14588 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:40223 every 8 connection(s)
I20260924 09:58:56.320749 14461 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/minicluster-data/ts-1/data/info.pb
I20260924 09:58:56.330739 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu as pid 14461
I20260924 09:58:56.330925 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/minicluster-data/ts-1/wal/instance
I20260924 09:58:56.331434 388 external_mini_cluster.cc:1428] Setting key 258f6455a7b5e7f674a90a12f37047b2
Sep 24 09:58:56 dist-test-slave-w8d2 krb5kdc[14221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243936, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 09:58:56.340387 14457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:58:56.323052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:50513 (local address 127.0.97.62:46551)
0924 09:58:56.323311 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0924 09:58:56.323317 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 09:58:56.324692 (+ 1375us) server_negotiation.cc:407] Connection header received
0924 09:58:56.326426 (+ 1734us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:58:56.326431 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:58:56.326503 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:58:56.326592 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:58:56.328056 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:58:56.328811 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:58:56.330309 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:58:56.330878 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:58:56.331012 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:58:56.334448 (+ 3436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:58:56.334455 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:58:56.334460 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:58:56.334496 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:58:56.337534 (+ 3038us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:58:56.338187 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:58:56.338192 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:58:56.338195 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:58:56.338341 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:58:56.338700 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:58:56.338704 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:58:56.338717 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:58:56.339057 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:58:56.339185 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0924 09:58:56.339851 (+ 666us) server_negotiation.cc:299] Negotiation successful
0924 09:58:56.340068 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260924 09:58:56.341280 14591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:58:56.323635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46551 (local address 127.0.97.2:50513)
0924 09:58:56.324486 (+ 851us) negotiation.cc:107] Waiting for socket to connect
0924 09:58:56.324535 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0924 09:58:56.326148 (+ 1613us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:58:56.326799 (+ 651us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:58:56.326807 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:58:56.327313 (+ 506us) client_negotiation.cc:189] Negotiated authn=SASL
0924 09:58:56.327890 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:58:56.327908 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:58:56.328935 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:58:56.328939 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:58:56.330200 (+ 1261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:58:56.330212 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:58:56.331092 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:58:56.331097 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:58:56.331338 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:58:56.331962 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 09:58:56.331984 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 09:58:56.334204 (+ 2220us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 09:58:56.337664 (+ 3460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 09:58:56.337670 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 09:58:56.337682 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0924 09:58:56.338076 (+ 394us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 09:58:56.338432 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 09:58:56.338435 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 09:58:56.338437 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 09:58:56.338609 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 09:58:56.339176 (+ 567us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 09:58:56.339182 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 09:58:56.339527 (+ 345us) client_negotiation.cc:769] Sending connection context
0924 09:58:56.339744 (+ 217us) client_negotiation.cc:240] Negotiation successful
0924 09:58:56.340125 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":579,"thread_start_us":286,"threads_started":1}
I20260924 09:58:56.342873 14589 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46551
I20260924 09:58:56.343159 14589 heartbeater.cc:464] Registering TS with master...
I20260924 09:58:56.343865 14589 heartbeater.cc:511] Master 127.0.97.62:46551 requested a full tablet report, sending...
I20260924 09:58:56.345238 14266 ts_manager.cc:194] Registered new tserver with Master: d540747f7bab4b0aa074cda755e1642d (127.0.97.2:40223)
I20260924 09:58:56.346107 14266 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:50513
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260924 09:58:56.378156 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
/tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46551
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 09:58:56.682605 14594 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 09:58:56.682885 14594 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 09:58:56.683002 14594 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 09:58:56.690037 14594 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 09:58:56.690152 14594 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 09:58:56.690315 14594 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260924 09:58:56.700739 14594 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46551
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 09:58:56.701941 14594 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 09:58:56.703176 14594 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 09:58:56.715598 14599 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:58:56.715564 14600 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:58:56.715639 14602 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 09:58:56.716532 14594 server_base.cc:1034] running on GCE node
I20260924 09:58:56.717101 14594 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 09:58:56.717949 14594 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 09:58:56.719138 14594 hybrid_clock.cc:648] HybridClock initialized: now 1790243936719095 us; error 35 us; skew 500 ppm
I20260924 09:58:56.719442 14594 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 09:58:56 dist-test-slave-w8d2 krb5kdc[14221](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243936, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 09:58:56.722364 14594 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260924 09:58:56.723732 14594 webserver.cc:460] Webserver started at http://127.0.97.3:41853/ using document root <none> and password file <none>
I20260924 09:58:56.724373 14594 fs_manager.cc:359] Metadata directory not provided
I20260924 09:58:56.724478 14594 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 09:58:56.724743 14594 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 09:58:56.727602 14594 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/minicluster-data/ts-2/data/instance:
uuid: "df6733a6ac4a4b7b87437920eaa15585"
format_stamp: "Formatted at 2026-09-24 09:58:56 on dist-test-slave-w8d2"
server_key: "eb52ee37941673b737f37b0fa0e5abe5"
server_key_iv: "2116980e992b0b4705d1395bb478eccc"
server_key_version: "encryptionkey@0"
I20260924 09:58:56.728324 14594 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/minicluster-data/ts-2/wal/instance:
uuid: "df6733a6ac4a4b7b87437920eaa15585"
format_stamp: "Formatted at 2026-09-24 09:58:56 on dist-test-slave-w8d2"
server_key: "eb52ee37941673b737f37b0fa0e5abe5"
server_key_iv: "2116980e992b0b4705d1395bb478eccc"
server_key_version: "encryptionkey@0"
I20260924 09:58:56.733134 14594 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 09:58:56.736531 14609 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 09:58:56.737799 14594 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 09:58:56.737937 14594 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/minicluster-data/ts-2/wal
uuid: "df6733a6ac4a4b7b87437920eaa15585"
format_stamp: "Formatted at 2026-09-24 09:58:56 on dist-test-slave-w8d2"
server_key: "eb52ee37941673b737f37b0fa0e5abe5"
server_key_iv: "2116980e992b0b4705d1395bb478eccc"
server_key_version: "encryptionkey@0"
I20260924 09:58:56.738107 14594 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 09:58:56.759444 14594 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 09:58:56.763505 14594 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 09:58:56.763792 14594 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 09:58:56.764706 14594 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 09:58:56.765942 14594 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 09:58:56.766013 14594 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 09:58:56.766137 14594 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 09:58:56.766187 14594 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 09:58:56.794881 14594 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:39885
I20260924 09:58:56.794924 14721 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:39885 every 8 connection(s)
I20260924 09:58:56.796188 14594 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/minicluster-data/ts-2/data/info.pb
I20260924 09:58:56.797482 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu as pid 14594
I20260924 09:58:56.797632 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/minicluster-data/ts-2/wal/instance
I20260924 09:58:56.798099 388 external_mini_cluster.cc:1428] Setting key c178c41dbe3c599d1dd951258acf81cf
Sep 24 09:58:56 dist-test-slave-w8d2 krb5kdc[14221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243936, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 09:58:56.814996 14457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:58:56.798637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:60069 (local address 127.0.97.62:46551)
0924 09:58:56.798858 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0924 09:58:56.798865 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 09:58:56.800162 (+ 1297us) server_negotiation.cc:407] Connection header received
0924 09:58:56.802183 (+ 2021us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:58:56.802188 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:58:56.802271 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:58:56.802355 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:58:56.803794 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:58:56.804558 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:58:56.806052 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:58:56.806466 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:58:56.806566 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:58:56.809579 (+ 3013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:58:56.809585 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:58:56.809590 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:58:56.809627 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:58:56.811999 (+ 2372us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:58:56.812832 (+ 833us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:58:56.812836 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:58:56.812840 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:58:56.812978 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:58:56.813514 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:58:56.813518 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:58:56.813520 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:58:56.813778 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:58:56.813886 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0924 09:58:56.814598 (+ 712us) server_negotiation.cc:299] Negotiation successful
0924 09:58:56.814759 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260924 09:58:56.815979 14724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:58:56.799226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46551 (local address 127.0.97.3:60069)
0924 09:58:56.800027 (+ 801us) negotiation.cc:107] Waiting for socket to connect
0924 09:58:56.800068 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 09:58:56.801903 (+ 1835us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:58:56.802582 (+ 679us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:58:56.802590 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:58:56.803063 (+ 473us) client_negotiation.cc:189] Negotiated authn=SASL
0924 09:58:56.803640 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:58:56.803658 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:58:56.804690 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:58:56.804694 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:58:56.805918 (+ 1224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:58:56.805930 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:58:56.806657 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:58:56.806664 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:58:56.806872 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:58:56.807441 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 09:58:56.807463 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 09:58:56.809392 (+ 1929us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 09:58:56.812126 (+ 2734us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 09:58:56.812135 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 09:58:56.812149 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 09:58:56.812712 (+ 563us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 09:58:56.813106 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 09:58:56.813111 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 09:58:56.813114 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 09:58:56.813392 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 09:58:56.813950 (+ 558us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 09:58:56.813956 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 09:58:56.814264 (+ 308us) client_negotiation.cc:769] Sending connection context
0924 09:58:56.814508 (+ 244us) client_negotiation.cc:240] Negotiation successful
0924 09:58:56.814867 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":292,"threads_started":1}
I20260924 09:58:56.817189 14722 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46551
I20260924 09:58:56.817479 14722 heartbeater.cc:464] Registering TS with master...
I20260924 09:58:56.818122 14722 heartbeater.cc:511] Master 127.0.97.62:46551 requested a full tablet report, sending...
I20260924 09:58:56.819432 14266 ts_manager.cc:194] Registered new tserver with Master: df6733a6ac4a4b7b87437920eaa15585 (127.0.97.3:39885)
I20260924 09:58:56.820209 14266 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:60069
I20260924 09:58:56.825417 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 09:58:56.842490 14457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:58:56.830895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47738 (local address 127.0.97.62:46551)
0924 09:58:56.831101 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0924 09:58:56.831107 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 09:58:56.831415 (+ 308us) server_negotiation.cc:407] Connection header received
0924 09:58:56.831548 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:58:56.831551 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:58:56.831612 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:58:56.831687 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:58:56.832978 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:58:56.833672 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:58:56.834937 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:58:56.835426 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:58:56.835536 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:58:56.837736 (+ 2200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:58:56.837741 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:58:56.837744 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:58:56.837775 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:58:56.840035 (+ 2260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:58:56.840654 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:58:56.840660 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:58:56.840664 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:58:56.840785 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:58:56.841119 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:58:56.841125 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:58:56.841128 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:58:56.841422 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:58:56.841642 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0924 09:58:56.841984 (+ 342us) server_negotiation.cc:299] Negotiation successful
0924 09:58:56.842191 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 09:58:56.846033 14266 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47738:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 09:58:56.852499 14266 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 09:58:56.874022 14739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:58:56.865984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36858 (local address 127.0.97.2:40223)
0924 09:58:56.866488 (+ 504us) server_negotiation.cc:206] Beginning negotiation
0924 09:58:56.866496 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 09:58:56.866527 (+ 31us) server_negotiation.cc:407] Connection header received
0924 09:58:56.866619 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:58:56.866626 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:58:56.866830 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:58:56.866996 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 09:58:56.867647 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:58:56.869232 (+ 1585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:58:56.871262 (+ 2030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:58:56.872873 (+ 1611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:58:56.873009 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:58:56.873203 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0924 09:58:56.873546 (+ 343us) server_negotiation.cc:299] Negotiation successful
0924 09:58:56.873698 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":346,"thread_start_us":230,"threads_started":1}
I20260924 09:58:56.874089 14735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:58:56.864174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:40223 (local address 127.0.0.1:36858)
0924 09:58:56.866063 (+ 1889us) negotiation.cc:107] Waiting for socket to connect
0924 09:58:56.866084 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 09:58:56.866190 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:58:56.867017 (+ 827us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:58:56.867024 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:58:56.867053 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 09:58:56.867423 (+ 370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:58:56.867434 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:58:56.869367 (+ 1933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:58:56.869375 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:58:56.871095 (+ 1720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:58:56.871107 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:58:56.873078 (+ 1971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:58:56.873085 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:58:56.873313 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:58:56.873383 (+ 70us) client_negotiation.cc:769] Sending connection context
0924 09:58:56.873488 (+ 105us) client_negotiation.cc:240] Negotiation successful
0924 09:58:56.873670 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1777,"mutex_wait_us":75,"thread_start_us":1643,"threads_started":1}
I20260924 09:58:56.875047 14737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:58:56.864929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44502 (local address 127.0.97.1:36377)
0924 09:58:56.866824 (+ 1895us) server_negotiation.cc:206] Beginning negotiation
0924 09:58:56.866833 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 09:58:56.866861 (+ 28us) server_negotiation.cc:407] Connection header received
0924 09:58:56.866952 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:58:56.866959 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:58:56.867178 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:58:56.867370 (+ 192us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 09:58:56.867987 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:58:56.869422 (+ 1435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:58:56.871927 (+ 2505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:58:56.873005 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:58:56.873120 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:58:56.873326 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0924 09:58:56.874558 (+ 1232us) server_negotiation.cc:299] Negotiation successful
0924 09:58:56.874740 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1729,"thread_start_us":1577,"threads_started":1}
I20260924 09:58:56.875415 14734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:58:56.863891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36377 (local address 127.0.0.1:44502)
0924 09:58:56.864807 (+ 916us) negotiation.cc:107] Waiting for socket to connect
0924 09:58:56.864844 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 09:58:56.864989 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:58:56.867395 (+ 2406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:58:56.867399 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:58:56.867427 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 09:58:56.867789 (+ 362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:58:56.867800 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:58:56.869539 (+ 1739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:58:56.869547 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:58:56.871740 (+ 2193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:58:56.871755 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:58:56.873392 (+ 1637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:58:56.873398 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:58:56.873570 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:58:56.873600 (+ 30us) client_negotiation.cc:769] Sending connection context
0924 09:58:56.873676 (+ 76us) client_negotiation.cc:240] Negotiation successful
0924 09:58:56.873802 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":683,"mutex_wait_us":44,"thread_start_us":554,"threads_started":1}
I20260924 09:58:56.876111 14736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:58:56.864501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:39885 (local address 127.0.0.1:60136)
0924 09:58:56.868348 (+ 3847us) negotiation.cc:107] Waiting for socket to connect
0924 09:58:56.868368 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 09:58:56.868479 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:58:56.869085 (+ 606us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:58:56.869089 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:58:56.869110 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 09:58:56.869507 (+ 397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:58:56.869519 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:58:56.871402 (+ 1883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:58:56.871406 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:58:56.873427 (+ 2021us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:58:56.873440 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:58:56.875065 (+ 1625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:58:56.875071 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:58:56.875224 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:58:56.875297 (+ 73us) client_negotiation.cc:769] Sending connection context
0924 09:58:56.875403 (+ 106us) client_negotiation.cc:240] Negotiation successful
0924 09:58:56.875793 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3721,"mutex_wait_us":3,"thread_start_us":4324,"threads_started":1}
I20260924 09:58:56.876444 14738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:58:56.865484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60136 (local address 127.0.97.3:39885)
0924 09:58:56.867979 (+ 2495us) server_negotiation.cc:206] Beginning negotiation
0924 09:58:56.867987 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 09:58:56.868591 (+ 604us) server_negotiation.cc:407] Connection header received
0924 09:58:56.868686 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:58:56.868693 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:58:56.868907 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:58:56.869084 (+ 177us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 09:58:56.869694 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:58:56.871270 (+ 1576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:58:56.873595 (+ 2325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:58:56.874873 (+ 1278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:58:56.874995 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:58:56.875193 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0924 09:58:56.875684 (+ 491us) server_negotiation.cc:299] Negotiation successful
0924 09:58:56.875856 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2350,"thread_start_us":2209,"threads_started":1}
I20260924 09:58:56.877921 14524 tablet_service.cc:1467] Processing CreateTablet for tablet f5d01dae68ef461c9e8a12d28a3f5cf7 (DEFAULT_TABLE table=test-table [id=ce320c5d5b8044b8b102a21d77189337]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 09:58:56.878798 14390 tablet_service.cc:1467] Processing CreateTablet for tablet f5d01dae68ef461c9e8a12d28a3f5cf7 (DEFAULT_TABLE table=test-table [id=ce320c5d5b8044b8b102a21d77189337]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 09:58:56.879345 14524 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5d01dae68ef461c9e8a12d28a3f5cf7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 09:58:56.879073 14657 tablet_service.cc:1467] Processing CreateTablet for tablet f5d01dae68ef461c9e8a12d28a3f5cf7 (DEFAULT_TABLE table=test-table [id=ce320c5d5b8044b8b102a21d77189337]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 09:58:56.880096 14390 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5d01dae68ef461c9e8a12d28a3f5cf7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 09:58:56.880131 14657 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5d01dae68ef461c9e8a12d28a3f5cf7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 09:58:56.881134 14455 heartbeater.cc:503] Master 127.0.97.62:46551 was elected leader, sending a full tablet report...
I20260924 09:58:56.891148 14740 tablet_bootstrap.cc:492] T f5d01dae68ef461c9e8a12d28a3f5cf7 P df6733a6ac4a4b7b87437920eaa15585: Bootstrap starting.
I20260924 09:58:56.893108 14741 tablet_bootstrap.cc:492] T f5d01dae68ef461c9e8a12d28a3f5cf7 P d540747f7bab4b0aa074cda755e1642d: Bootstrap starting.
I20260924 09:58:56.895138 14740 tablet_bootstrap.cc:654] T f5d01dae68ef461c9e8a12d28a3f5cf7 P df6733a6ac4a4b7b87437920eaa15585: Neither blocks nor log segments found. Creating new log.
I20260924 09:58:56.896327 14741 tablet_bootstrap.cc:654] T f5d01dae68ef461c9e8a12d28a3f5cf7 P d540747f7bab4b0aa074cda755e1642d: Neither blocks nor log segments found. Creating new log.
I20260924 09:58:56.896380 14740 log.cc:824] T f5d01dae68ef461c9e8a12d28a3f5cf7 P df6733a6ac4a4b7b87437920eaa15585: Log is configured to *not* fsync() on all Append() calls
I20260924 09:58:56.897481 14741 log.cc:824] T f5d01dae68ef461c9e8a12d28a3f5cf7 P d540747f7bab4b0aa074cda755e1642d: Log is configured to *not* fsync() on all Append() calls
I20260924 09:58:56.897665 14742 tablet_bootstrap.cc:492] T f5d01dae68ef461c9e8a12d28a3f5cf7 P fc45f069420445869c886224da489dc6: Bootstrap starting.
I20260924 09:58:56.899682 14740 tablet_bootstrap.cc:492] T f5d01dae68ef461c9e8a12d28a3f5cf7 P df6733a6ac4a4b7b87437920eaa15585: No bootstrap required, opened a new log
I20260924 09:58:56.899959 14740 ts_tablet_manager.cc:1397] T f5d01dae68ef461c9e8a12d28a3f5cf7 P df6733a6ac4a4b7b87437920eaa15585: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260924 09:58:56.900152 14741 tablet_bootstrap.cc:492] T f5d01dae68ef461c9e8a12d28a3f5cf7 P d540747f7bab4b0aa074cda755e1642d: No bootstrap required, opened a new log
I20260924 09:58:56.900395 14741 ts_tablet_manager.cc:1397] T f5d01dae68ef461c9e8a12d28a3f5cf7 P d540747f7bab4b0aa074cda755e1642d: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260924 09:58:56.901811 14742 tablet_bootstrap.cc:654] T f5d01dae68ef461c9e8a12d28a3f5cf7 P fc45f069420445869c886224da489dc6: Neither blocks nor log segments found. Creating new log.
I20260924 09:58:56.902841 14742 log.cc:824] T f5d01dae68ef461c9e8a12d28a3f5cf7 P fc45f069420445869c886224da489dc6: Log is configured to *not* fsync() on all Append() calls
I20260924 09:58:56.909222 14742 tablet_bootstrap.cc:492] T f5d01dae68ef461c9e8a12d28a3f5cf7 P fc45f069420445869c886224da489dc6: No bootstrap required, opened a new log
I20260924 09:58:56.909456 14742 ts_tablet_manager.cc:1397] T f5d01dae68ef461c9e8a12d28a3f5cf7 P fc45f069420445869c886224da489dc6: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.004s
I20260924 09:58:56.910120 14740 raft_consensus.cc:348] T f5d01dae68ef461c9e8a12d28a3f5cf7 P df6733a6ac4a4b7b87437920eaa15585 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc45f069420445869c886224da489dc6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36377 } } peers { permanent_uuid: "d540747f7bab4b0aa074cda755e1642d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40223 } } peers { permanent_uuid: "df6733a6ac4a4b7b87437920eaa15585" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39885 } }
I20260924 09:58:56.910109 14741 raft_consensus.cc:348] T f5d01dae68ef461c9e8a12d28a3f5cf7 P d540747f7bab4b0aa074cda755e1642d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc45f069420445869c886224da489dc6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36377 } } peers { permanent_uuid: "d540747f7bab4b0aa074cda755e1642d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40223 } } peers { permanent_uuid: "df6733a6ac4a4b7b87437920eaa15585" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39885 } }
I20260924 09:58:56.910516 14741 raft_consensus.cc:374] T f5d01dae68ef461c9e8a12d28a3f5cf7 P d540747f7bab4b0aa074cda755e1642d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 09:58:56.910530 14740 raft_consensus.cc:374] T f5d01dae68ef461c9e8a12d28a3f5cf7 P df6733a6ac4a4b7b87437920eaa15585 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 09:58:56.910604 14740 raft_consensus.cc:729] T f5d01dae68ef461c9e8a12d28a3f5cf7 P df6733a6ac4a4b7b87437920eaa15585 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df6733a6ac4a4b7b87437920eaa15585, State: Initialized, Role: FOLLOWER
I20260924 09:58:56.910604 14741 raft_consensus.cc:729] T f5d01dae68ef461c9e8a12d28a3f5cf7 P d540747f7bab4b0aa074cda755e1642d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d540747f7bab4b0aa074cda755e1642d, State: Initialized, Role: FOLLOWER
I20260924 09:58:56.911180 14740 consensus_queue.cc:260] T f5d01dae68ef461c9e8a12d28a3f5cf7 P df6733a6ac4a4b7b87437920eaa15585 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc45f069420445869c886224da489dc6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36377 } } peers { permanent_uuid: "d540747f7bab4b0aa074cda755e1642d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40223 } } peers { permanent_uuid: "df6733a6ac4a4b7b87437920eaa15585" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39885 } }
I20260924 09:58:56.911183 14741 consensus_queue.cc:260] T f5d01dae68ef461c9e8a12d28a3f5cf7 P d540747f7bab4b0aa074cda755e1642d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc45f069420445869c886224da489dc6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36377 } } peers { permanent_uuid: "d540747f7bab4b0aa074cda755e1642d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40223 } } peers { permanent_uuid: "df6733a6ac4a4b7b87437920eaa15585" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39885 } }
I20260924 09:58:56.912238 14722 heartbeater.cc:503] Master 127.0.97.62:46551 was elected leader, sending a full tablet report...
I20260924 09:58:56.915880 14589 heartbeater.cc:503] Master 127.0.97.62:46551 was elected leader, sending a full tablet report...
I20260924 09:58:56.916597 14741 ts_tablet_manager.cc:1428] T f5d01dae68ef461c9e8a12d28a3f5cf7 P d540747f7bab4b0aa074cda755e1642d: Time spent starting tablet: real 0.016s user 0.012s sys 0.004s
I20260924 09:58:56.917436 14740 ts_tablet_manager.cc:1428] T f5d01dae68ef461c9e8a12d28a3f5cf7 P df6733a6ac4a4b7b87437920eaa15585: Time spent starting tablet: real 0.017s user 0.013s sys 0.000s
I20260924 09:58:56.917969 14742 raft_consensus.cc:348] T f5d01dae68ef461c9e8a12d28a3f5cf7 P fc45f069420445869c886224da489dc6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc45f069420445869c886224da489dc6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36377 } } peers { permanent_uuid: "d540747f7bab4b0aa074cda755e1642d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40223 } } peers { permanent_uuid: "df6733a6ac4a4b7b87437920eaa15585" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39885 } }
I20260924 09:58:56.918375 14742 raft_consensus.cc:374] T f5d01dae68ef461c9e8a12d28a3f5cf7 P fc45f069420445869c886224da489dc6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 09:58:56.918470 14742 raft_consensus.cc:729] T f5d01dae68ef461c9e8a12d28a3f5cf7 P fc45f069420445869c886224da489dc6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc45f069420445869c886224da489dc6, State: Initialized, Role: FOLLOWER
I20260924 09:58:56.919027 14742 consensus_queue.cc:260] T f5d01dae68ef461c9e8a12d28a3f5cf7 P fc45f069420445869c886224da489dc6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc45f069420445869c886224da489dc6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36377 } } peers { permanent_uuid: "d540747f7bab4b0aa074cda755e1642d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40223 } } peers { permanent_uuid: "df6733a6ac4a4b7b87437920eaa15585" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39885 } }
I20260924 09:58:56.920552 14742 ts_tablet_manager.cc:1428] T f5d01dae68ef461c9e8a12d28a3f5cf7 P fc45f069420445869c886224da489dc6: Time spent starting tablet: real 0.011s user 0.012s sys 0.000s
I20260924 09:58:57.036810 14747 raft_consensus.cc:482] T f5d01dae68ef461c9e8a12d28a3f5cf7 P d540747f7bab4b0aa074cda755e1642d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 09:58:57.037081 14747 raft_consensus.cc:504] T f5d01dae68ef461c9e8a12d28a3f5cf7 P d540747f7bab4b0aa074cda755e1642d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc45f069420445869c886224da489dc6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36377 } } peers { permanent_uuid: "d540747f7bab4b0aa074cda755e1642d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40223 } } peers { permanent_uuid: "df6733a6ac4a4b7b87437920eaa15585" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39885 } }
I20260924 09:58:57.038612 14747 leader_election.cc:288] T f5d01dae68ef461c9e8a12d28a3f5cf7 P d540747f7bab4b0aa074cda755e1642d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fc45f069420445869c886224da489dc6 (127.0.97.1:36377), df6733a6ac4a4b7b87437920eaa15585 (127.0.97.3:39885)
I20260924 09:58:57.045851 14737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:58:57.038987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:41353 (local address 127.0.97.1:36377)
0924 09:58:57.039195 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0924 09:58:57.039201 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 09:58:57.039916 (+ 715us) server_negotiation.cc:407] Connection header received
0924 09:58:57.040093 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:58:57.040097 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:58:57.040155 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:58:57.040225 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 09:58:57.040810 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:58:57.041564 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:58:57.044263 (+ 2699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:58:57.045040 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:58:57.045142 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:58:57.045217 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 09:58:57.045475 (+ 258us) server_negotiation.cc:299] Negotiation successful
0924 09:58:57.045596 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260924 09:58:57.046134 14749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:58:57.038796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36377 (local address 127.0.97.2:41353)
0924 09:58:57.039808 (+ 1012us) negotiation.cc:107] Waiting for socket to connect
0924 09:58:57.039824 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 09:58:57.039960 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:58:57.040252 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:58:57.040256 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:58:57.040279 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 09:58:57.040669 (+ 390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:58:57.040682 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:58:57.041690 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:58:57.041695 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:58:57.044122 (+ 2427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:58:57.044133 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:58:57.045209 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:58:57.045213 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:58:57.045322 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:58:57.045345 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 09:58:57.045422 (+ 77us) client_negotiation.cc:240] Negotiation successful
0924 09:58:57.045553 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":895,"mutex_wait_us":56,"thread_start_us":695,"threads_started":1}
I20260924 09:58:57.047200 14750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:58:57.039018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:39885 (local address 127.0.97.2:36617)
0924 09:58:57.039899 (+ 881us) negotiation.cc:107] Waiting for socket to connect
0924 09:58:57.039913 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 09:58:57.040047 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:58:57.040336 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:58:57.040340 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:58:57.040376 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 09:58:57.040717 (+ 341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:58:57.040725 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:58:57.041731 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:58:57.041735 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:58:57.044120 (+ 2385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:58:57.044133 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:58:57.046528 (+ 2395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:58:57.046535 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:58:57.046710 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:58:57.046765 (+ 55us) client_negotiation.cc:769] Sending connection context
0924 09:58:57.046850 (+ 85us) client_negotiation.cc:240] Negotiation successful
0924 09:58:57.046938 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":785,"mutex_wait_us":117,"thread_start_us":508,"threads_started":1}
I20260924 09:58:57.047263 14738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:58:57.039098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:36617 (local address 127.0.97.3:39885)
0924 09:58:57.039324 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0924 09:58:57.039330 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 09:58:57.040000 (+ 670us) server_negotiation.cc:407] Connection header received
0924 09:58:57.040170 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:58:57.040173 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:58:57.040230 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:58:57.040303 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 09:58:57.040868 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:58:57.041626 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:58:57.045286 (+ 3660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:58:57.046157 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:58:57.046281 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:58:57.046392 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0924 09:58:57.046880 (+ 488us) server_negotiation.cc:299] Negotiation successful
0924 09:58:57.047019 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 09:58:57.047093 14410 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5d01dae68ef461c9e8a12d28a3f5cf7" candidate_uuid: "d540747f7bab4b0aa074cda755e1642d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc45f069420445869c886224da489dc6" is_pre_election: true
I20260924 09:58:57.047708 14410 raft_consensus.cc:2454] T f5d01dae68ef461c9e8a12d28a3f5cf7 P fc45f069420445869c886224da489dc6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d540747f7bab4b0aa074cda755e1642d in term 0.
W20260924 09:58:57.047755 14723 tablet.cc:2369] T f5d01dae68ef461c9e8a12d28a3f5cf7 P df6733a6ac4a4b7b87437920eaa15585: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 09:58:57.048343 14480 leader_election.cc:302] T f5d01dae68ef461c9e8a12d28a3f5cf7 P d540747f7bab4b0aa074cda755e1642d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d540747f7bab4b0aa074cda755e1642d, fc45f069420445869c886224da489dc6; no voters:
I20260924 09:58:57.048203 14677 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5d01dae68ef461c9e8a12d28a3f5cf7" candidate_uuid: "d540747f7bab4b0aa074cda755e1642d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df6733a6ac4a4b7b87437920eaa15585" is_pre_election: true
I20260924 09:58:57.048683 14747 raft_consensus.cc:2781] T f5d01dae68ef461c9e8a12d28a3f5cf7 P d540747f7bab4b0aa074cda755e1642d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 09:58:57.048702 14677 raft_consensus.cc:2454] T f5d01dae68ef461c9e8a12d28a3f5cf7 P df6733a6ac4a4b7b87437920eaa15585 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d540747f7bab4b0aa074cda755e1642d in term 0.
I20260924 09:58:57.048799 14747 raft_consensus.cc:482] T f5d01dae68ef461c9e8a12d28a3f5cf7 P d540747f7bab4b0aa074cda755e1642d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 09:58:57.048906 14747 raft_consensus.cc:3037] T f5d01dae68ef461c9e8a12d28a3f5cf7 P d540747f7bab4b0aa074cda755e1642d [term 0 FOLLOWER]: Advancing to term 1
I20260924 09:58:57.051067 14747 raft_consensus.cc:504] T f5d01dae68ef461c9e8a12d28a3f5cf7 P d540747f7bab4b0aa074cda755e1642d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc45f069420445869c886224da489dc6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36377 } } peers { permanent_uuid: "d540747f7bab4b0aa074cda755e1642d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40223 } } peers { permanent_uuid: "df6733a6ac4a4b7b87437920eaa15585" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39885 } }
I20260924 09:58:57.051654 14747 leader_election.cc:288] T f5d01dae68ef461c9e8a12d28a3f5cf7 P d540747f7bab4b0aa074cda755e1642d [CANDIDATE]: Term 1 election: Requested vote from peers fc45f069420445869c886224da489dc6 (127.0.97.1:36377), df6733a6ac4a4b7b87437920eaa15585 (127.0.97.3:39885)
I20260924 09:58:57.052057 14410 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5d01dae68ef461c9e8a12d28a3f5cf7" candidate_uuid: "d540747f7bab4b0aa074cda755e1642d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc45f069420445869c886224da489dc6"
I20260924 09:58:57.052083 14677 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5d01dae68ef461c9e8a12d28a3f5cf7" candidate_uuid: "d540747f7bab4b0aa074cda755e1642d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df6733a6ac4a4b7b87437920eaa15585"
I20260924 09:58:57.052251 14677 raft_consensus.cc:3037] T f5d01dae68ef461c9e8a12d28a3f5cf7 P df6733a6ac4a4b7b87437920eaa15585 [term 0 FOLLOWER]: Advancing to term 1
I20260924 09:58:57.052253 14410 raft_consensus.cc:3037] T f5d01dae68ef461c9e8a12d28a3f5cf7 P fc45f069420445869c886224da489dc6 [term 0 FOLLOWER]: Advancing to term 1
I20260924 09:58:57.054278 14677 raft_consensus.cc:2454] T f5d01dae68ef461c9e8a12d28a3f5cf7 P df6733a6ac4a4b7b87437920eaa15585 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d540747f7bab4b0aa074cda755e1642d in term 1.
I20260924 09:58:57.054584 14410 raft_consensus.cc:2454] T f5d01dae68ef461c9e8a12d28a3f5cf7 P fc45f069420445869c886224da489dc6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d540747f7bab4b0aa074cda755e1642d in term 1.
I20260924 09:58:57.054620 14479 leader_election.cc:302] T f5d01dae68ef461c9e8a12d28a3f5cf7 P d540747f7bab4b0aa074cda755e1642d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d540747f7bab4b0aa074cda755e1642d, df6733a6ac4a4b7b87437920eaa15585; no voters:
I20260924 09:58:57.054916 14747 raft_consensus.cc:2781] T f5d01dae68ef461c9e8a12d28a3f5cf7 P d540747f7bab4b0aa074cda755e1642d [term 1 FOLLOWER]: Leader election won for term 1
I20260924 09:58:57.055351 14747 raft_consensus.cc:686] T f5d01dae68ef461c9e8a12d28a3f5cf7 P d540747f7bab4b0aa074cda755e1642d [term 1 LEADER]: Becoming Leader. State: Replica: d540747f7bab4b0aa074cda755e1642d, State: Running, Role: LEADER
I20260924 09:58:57.055802 14747 consensus_queue.cc:237] T f5d01dae68ef461c9e8a12d28a3f5cf7 P d540747f7bab4b0aa074cda755e1642d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc45f069420445869c886224da489dc6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36377 } } peers { permanent_uuid: "d540747f7bab4b0aa074cda755e1642d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40223 } } peers { permanent_uuid: "df6733a6ac4a4b7b87437920eaa15585" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39885 } }
I20260924 09:58:57.060067 14265 catalog_manager.cc:5495] T f5d01dae68ef461c9e8a12d28a3f5cf7 P d540747f7bab4b0aa074cda755e1642d reported cstate change: term changed from 0 to 1, leader changed from <none> to d540747f7bab4b0aa074cda755e1642d (127.0.97.2). New cstate: current_term: 1 leader_uuid: "d540747f7bab4b0aa074cda755e1642d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc45f069420445869c886224da489dc6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36377 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d540747f7bab4b0aa074cda755e1642d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40223 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df6733a6ac4a4b7b87437920eaa15585" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39885 } health_report { overall_health: UNKNOWN } } }
W20260924 09:58:57.072532 14590 tablet.cc:2369] T f5d01dae68ef461c9e8a12d28a3f5cf7 P d540747f7bab4b0aa074cda755e1642d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 09:58:57.102495 14456 tablet.cc:2369] T f5d01dae68ef461c9e8a12d28a3f5cf7 P fc45f069420445869c886224da489dc6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 09:58:57.109762 14739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:58:57.105112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36862 (local address 127.0.97.2:40223)
0924 09:58:57.105294 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0924 09:58:57.105299 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 09:58:57.105316 (+ 17us) server_negotiation.cc:407] Connection header received
0924 09:58:57.105476 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:58:57.105479 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:58:57.105558 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:58:57.105675 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 09:58:57.105952 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:58:57.106632 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:58:57.107896 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:58:57.108281 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:58:57.108366 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:58:57.108761 (+ 395us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 09:58:57.109289 (+ 528us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 09:58:57.109389 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0924 09:58:57.109462 (+ 73us) server_negotiation.cc:299] Negotiation successful
0924 09:58:57.109523 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260924 09:58:57.117100 14677 raft_consensus.cc:1261] T f5d01dae68ef461c9e8a12d28a3f5cf7 P df6733a6ac4a4b7b87437920eaa15585 [term 1 FOLLOWER]: Refusing update from remote peer d540747f7bab4b0aa074cda755e1642d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 09:58:57.117112 14410 raft_consensus.cc:1261] T f5d01dae68ef461c9e8a12d28a3f5cf7 P fc45f069420445869c886224da489dc6 [term 1 FOLLOWER]: Refusing update from remote peer d540747f7bab4b0aa074cda755e1642d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 09:58:57.117933 14751 consensus_queue.cc:1035] T f5d01dae68ef461c9e8a12d28a3f5cf7 P d540747f7bab4b0aa074cda755e1642d [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6733a6ac4a4b7b87437920eaa15585" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 09:58:57.118252 14747 consensus_queue.cc:1035] T f5d01dae68ef461c9e8a12d28a3f5cf7 P d540747f7bab4b0aa074cda755e1642d [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc45f069420445869c886224da489dc6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 09:58:57.128316 14754 mvcc.cc:205] Tried to move back new op lower bound from 7332839166422511616 to 7332839166183895040. Current Snapshot: MvccSnapshot[applied={T|T < 7332839166422511616}]
I20260924 09:58:57.129523 14755 mvcc.cc:205] Tried to move back new op lower bound from 7332839166422511616 to 7332839166183895040. Current Snapshot: MvccSnapshot[applied={T|T < 7332839166422511616}]
I20260924 09:58:57.140267 14265 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47738:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 09:58:57.140571 14265 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47738:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 09:58:57.144143 14265 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 76cdda25f78f4f8dab4b5e8cdf63a2b7: Sending DeleteTablet for 3 replicas of tablet f5d01dae68ef461c9e8a12d28a3f5cf7
I20260924 09:58:57.145210 14524 tablet_service.cc:1514] Processing DeleteTablet for tablet f5d01dae68ef461c9e8a12d28a3f5cf7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 09:58:57 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:36858
I20260924 09:58:57.145205 14390 tablet_service.cc:1514] Processing DeleteTablet for tablet f5d01dae68ef461c9e8a12d28a3f5cf7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 09:58:57 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:44502
I20260924 09:58:57.145617 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu pid 14327
I20260924 09:58:57.146119 14657 tablet_service.cc:1514] Processing DeleteTablet for tablet f5d01dae68ef461c9e8a12d28a3f5cf7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 09:58:57 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:60136
I20260924 09:58:57.146507 14765 tablet_replica.cc:330] T f5d01dae68ef461c9e8a12d28a3f5cf7 P d540747f7bab4b0aa074cda755e1642d: stopping tablet replica
I20260924 09:58:57.146888 14765 raft_consensus.cc:2229] T f5d01dae68ef461c9e8a12d28a3f5cf7 P d540747f7bab4b0aa074cda755e1642d [term 1 LEADER]: Raft consensus shutting down.
I20260924 09:58:57.147531 14765 raft_consensus.cc:2258] T f5d01dae68ef461c9e8a12d28a3f5cf7 P d540747f7bab4b0aa074cda755e1642d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 09:58:57.148248 14764 tablet_replica.cc:330] T f5d01dae68ef461c9e8a12d28a3f5cf7 P fc45f069420445869c886224da489dc6: stopping tablet replica
I20260924 09:58:57.148648 14764 raft_consensus.cc:2229] T f5d01dae68ef461c9e8a12d28a3f5cf7 P fc45f069420445869c886224da489dc6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 09:58:57.149125 14764 raft_consensus.cc:2258] T f5d01dae68ef461c9e8a12d28a3f5cf7 P fc45f069420445869c886224da489dc6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 09:58:57.151377 14764 ts_tablet_manager.cc:1905] T f5d01dae68ef461c9e8a12d28a3f5cf7 P fc45f069420445869c886224da489dc6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 09:58:57.155484 14765 ts_tablet_manager.cc:1905] T f5d01dae68ef461c9e8a12d28a3f5cf7 P d540747f7bab4b0aa074cda755e1642d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 09:58:57.158216 14764 ts_tablet_manager.cc:1918] T f5d01dae68ef461c9e8a12d28a3f5cf7 P fc45f069420445869c886224da489dc6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 09:58:57.158437 14764 log.cc:1196] T f5d01dae68ef461c9e8a12d28a3f5cf7 P fc45f069420445869c886224da489dc6: Deleting WAL directory at /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/minicluster-data/ts-0/wal/wals/f5d01dae68ef461c9e8a12d28a3f5cf7
I20260924 09:58:57.159044 14764 ts_tablet_manager.cc:1939] T f5d01dae68ef461c9e8a12d28a3f5cf7 P fc45f069420445869c886224da489dc6: Deleting consensus metadata
I20260924 09:58:57.161032 14253 catalog_manager.cc:4841] TS fc45f069420445869c886224da489dc6 (127.0.97.1:36377): tablet f5d01dae68ef461c9e8a12d28a3f5cf7 (table test-table [id=ce320c5d5b8044b8b102a21d77189337]) successfully deleted
I20260924 09:58:57.162753 14767 tablet_replica.cc:330] T f5d01dae68ef461c9e8a12d28a3f5cf7 P df6733a6ac4a4b7b87437920eaa15585: stopping tablet replica
I20260924 09:58:57.163166 14767 raft_consensus.cc:2229] T f5d01dae68ef461c9e8a12d28a3f5cf7 P df6733a6ac4a4b7b87437920eaa15585 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 09:58:57.163687 14767 raft_consensus.cc:2258] T f5d01dae68ef461c9e8a12d28a3f5cf7 P df6733a6ac4a4b7b87437920eaa15585 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 09:58:57.167277 14765 ts_tablet_manager.cc:1918] T f5d01dae68ef461c9e8a12d28a3f5cf7 P d540747f7bab4b0aa074cda755e1642d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 09:58:57.167450 14765 log.cc:1196] T f5d01dae68ef461c9e8a12d28a3f5cf7 P d540747f7bab4b0aa074cda755e1642d: Deleting WAL directory at /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/minicluster-data/ts-1/wal/wals/f5d01dae68ef461c9e8a12d28a3f5cf7
I20260924 09:58:57.169602 14767 ts_tablet_manager.cc:1905] T f5d01dae68ef461c9e8a12d28a3f5cf7 P df6733a6ac4a4b7b87437920eaa15585: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 24 09:58:57 dist-test-slave-w8d2 krb5kdc[14221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243934, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260924 09:58:57.171607 14765 ts_tablet_manager.cc:1939] T f5d01dae68ef461c9e8a12d28a3f5cf7 P d540747f7bab4b0aa074cda755e1642d: Deleting consensus metadata
I20260924 09:58:57.173365 14251 catalog_manager.cc:4841] TS d540747f7bab4b0aa074cda755e1642d (127.0.97.2:40223): tablet f5d01dae68ef461c9e8a12d28a3f5cf7 (table test-table [id=ce320c5d5b8044b8b102a21d77189337]) successfully deleted
I20260924 09:58:57.176263 14767 ts_tablet_manager.cc:1918] T f5d01dae68ef461c9e8a12d28a3f5cf7 P df6733a6ac4a4b7b87437920eaa15585: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 09:58:57.176415 14767 log.cc:1196] T f5d01dae68ef461c9e8a12d28a3f5cf7 P df6733a6ac4a4b7b87437920eaa15585: Deleting WAL directory at /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/minicluster-data/ts-2/wal/wals/f5d01dae68ef461c9e8a12d28a3f5cf7
I20260924 09:58:57.176889 14767 ts_tablet_manager.cc:1939] T f5d01dae68ef461c9e8a12d28a3f5cf7 P df6733a6ac4a4b7b87437920eaa15585: Deleting consensus metadata
I20260924 09:58:57.177106 14737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:58:57.146736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44504 (local address 127.0.97.1:36377)
0924 09:58:57.146900 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0924 09:58:57.146906 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 09:58:57.162443 (+ 15537us) server_negotiation.cc:407] Connection header received
0924 09:58:57.162608 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:58:57.162613 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:58:57.162694 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:58:57.162773 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:58:57.164329 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:58:57.165126 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:58:57.166538 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:58:57.166938 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:58:57.167045 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:58:57.171150 (+ 4105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:58:57.171163 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:58:57.171176 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:58:57.171208 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:58:57.174580 (+ 3372us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:58:57.175216 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:58:57.175224 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:58:57.175228 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:58:57.175394 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:58:57.175746 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:58:57.175751 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:58:57.175755 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:58:57.176088 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:58:57.176429 (+ 341us) server_negotiation.cc:1035] Waiting for connection context
0924 09:58:57.176657 (+ 228us) server_negotiation.cc:299] Negotiation successful
0924 09:58:57.176819 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 09:58:57.178216 14450 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 09:58:57.178686 14252 catalog_manager.cc:4841] TS df6733a6ac4a4b7b87437920eaa15585 (127.0.97.3:39885): tablet f5d01dae68ef461c9e8a12d28a3f5cf7 (table test-table [id=ce320c5d5b8044b8b102a21d77189337]) successfully deleted
I20260924 09:58:57.266516 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu with pid 14327
I20260924 09:58:57.290859 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu pid 14461
Sep 24 09:58:57 dist-test-slave-w8d2 krb5kdc[14221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243934, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260924 09:58:57.306046 14739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:58:57.291670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36870 (local address 127.0.97.2:40223)
0924 09:58:57.291870 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0924 09:58:57.291876 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 09:58:57.291894 (+ 18us) server_negotiation.cc:407] Connection header received
0924 09:58:57.292039 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:58:57.292044 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:58:57.292115 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:58:57.292225 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:58:57.293475 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:58:57.294279 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:58:57.295676 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:58:57.296095 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:58:57.296212 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:58:57.300257 (+ 4045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:58:57.300274 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:58:57.300289 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:58:57.300323 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:58:57.303379 (+ 3056us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:58:57.304020 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:58:57.304029 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:58:57.304033 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:58:57.304209 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:58:57.304539 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:58:57.304548 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:58:57.304552 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:58:57.304921 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:58:57.305221 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0924 09:58:57.305478 (+ 257us) server_negotiation.cc:299] Negotiation successful
0924 09:58:57.305684 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 09:58:57.306744 14584 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 09:58:57.417752 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu with pid 14461
I20260924 09:58:57.433115 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu pid 14594
Sep 24 09:58:57 dist-test-slave-w8d2 krb5kdc[14221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243934, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260924 09:58:57.447077 14738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:58:57.433819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60152 (local address 127.0.97.3:39885)
0924 09:58:57.434011 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0924 09:58:57.434016 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 09:58:57.434034 (+ 18us) server_negotiation.cc:407] Connection header received
0924 09:58:57.434173 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:58:57.434177 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:58:57.434240 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:58:57.434328 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:58:57.435594 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:58:57.436349 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:58:57.437544 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:58:57.437911 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:58:57.437998 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:58:57.442090 (+ 4092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:58:57.442103 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:58:57.442119 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:58:57.442160 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:58:57.444905 (+ 2745us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:58:57.445497 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:58:57.445503 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:58:57.445512 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:58:57.445632 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:58:57.445894 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:58:57.445897 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:58:57.445899 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:58:57.446205 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:58:57.446404 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0924 09:58:57.446632 (+ 228us) server_negotiation.cc:299] Negotiation successful
0924 09:58:57.446780 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260924 09:58:57.447759 14717 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 09:58:57.533790 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu with pid 14594
I20260924 09:58:57.547021 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu pid 14234
I20260924 09:58:57.548132 14296 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 09:58:57.656448 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu with pid 14234
I20260924 09:58:57.673185 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
/tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46551
--webserver_interface=127.0.97.62
--webserver_port=38575
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:46551
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 09:58:57.979249 14772 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 09:58:57.979540 14772 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 09:58:57.979667 14772 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 09:58:57.986645 14772 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 09:58:57.986768 14772 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 09:58:57.986840 14772 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 09:58:57.986912 14772 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 09:58:57.986968 14772 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 09:58:57.996690 14772 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:46551
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46551
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=38575
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 09:58:57.997921 14772 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 09:58:57.999181 14772 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 09:58:58.019069 14779 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:58:58.019093 14778 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:58:58.019035 14781 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 09:58:58.020354 14772 server_base.cc:1034] running on GCE node
I20260924 09:58:58.021174 14772 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 09:58:58.022688 14772 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 09:58:58.023881 14772 hybrid_clock.cc:648] HybridClock initialized: now 1790243938023824 us; error 42 us; skew 500 ppm
I20260924 09:58:58.024240 14772 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 09:58:58 dist-test-slave-w8d2 krb5kdc[14221](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243938, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 09:58:58.027544 14772 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260924 09:58:58.029177 14772 webserver.cc:460] Webserver started at http://127.0.97.62:38575/ using document root <none> and password file <none>
I20260924 09:58:58.029888 14772 fs_manager.cc:359] Metadata directory not provided
I20260924 09:58:58.030014 14772 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 09:58:58.035761 14772 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260924 09:58:58.038551 14788 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 09:58:58.039906 14772 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 09:58:58.040064 14772 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/minicluster-data/master-0/wal
uuid: "76cdda25f78f4f8dab4b5e8cdf63a2b7"
format_stamp: "Formatted at 2026-09-24 09:58:55 on dist-test-slave-w8d2"
server_key: "455e524ee257bc375ab1961387ac0eca"
server_key_iv: "8c564b7cccd5eb633989057664a5c375"
server_key_version: "encryptionkey@0"
I20260924 09:58:58.041090 14772 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 09:58:58.064760 14772 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 09:58:58.069058 14772 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 09:58:58.069316 14772 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 09:58:58.086333 14772 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:46551
I20260924 09:58:58.086311 14839 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:46551 every 8 connection(s)
I20260924 09:58:58.087827 14772 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/minicluster-data/master-0/data/info.pb
I20260924 09:58:58.091444 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu as pid 14772
I20260924 09:58:58.092348 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
/tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:36377
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:46551
--webserver_port=44965
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 09:58:58.096025 14840 sys_catalog.cc:263] Verifying existing consensus state
I20260924 09:58:58.100596 14840 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76cdda25f78f4f8dab4b5e8cdf63a2b7: Bootstrap starting.
I20260924 09:58:58.120083 14840 log.cc:824] T 00000000000000000000000000000000 P 76cdda25f78f4f8dab4b5e8cdf63a2b7: Log is configured to *not* fsync() on all Append() calls
I20260924 09:58:58.132294 14840 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76cdda25f78f4f8dab4b5e8cdf63a2b7: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 09:58:58.132802 14840 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76cdda25f78f4f8dab4b5e8cdf63a2b7: Bootstrap complete.
I20260924 09:58:58.141654 14840 raft_consensus.cc:348] T 00000000000000000000000000000000 P 76cdda25f78f4f8dab4b5e8cdf63a2b7 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76cdda25f78f4f8dab4b5e8cdf63a2b7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46551 } }
I20260924 09:58:58.142453 14840 raft_consensus.cc:729] T 00000000000000000000000000000000 P 76cdda25f78f4f8dab4b5e8cdf63a2b7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76cdda25f78f4f8dab4b5e8cdf63a2b7, State: Initialized, Role: FOLLOWER
I20260924 09:58:58.143081 14840 consensus_queue.cc:260] T 00000000000000000000000000000000 P 76cdda25f78f4f8dab4b5e8cdf63a2b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76cdda25f78f4f8dab4b5e8cdf63a2b7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46551 } }
I20260924 09:58:58.143373 14840 raft_consensus.cc:388] T 00000000000000000000000000000000 P 76cdda25f78f4f8dab4b5e8cdf63a2b7 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 09:58:58.143517 14840 raft_consensus.cc:482] T 00000000000000000000000000000000 P 76cdda25f78f4f8dab4b5e8cdf63a2b7 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 09:58:58.143679 14840 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 76cdda25f78f4f8dab4b5e8cdf63a2b7 [term 1 FOLLOWER]: Advancing to term 2
I20260924 09:58:58.147630 14840 raft_consensus.cc:504] T 00000000000000000000000000000000 P 76cdda25f78f4f8dab4b5e8cdf63a2b7 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76cdda25f78f4f8dab4b5e8cdf63a2b7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46551 } }
I20260924 09:58:58.148101 14840 leader_election.cc:302] T 00000000000000000000000000000000 P 76cdda25f78f4f8dab4b5e8cdf63a2b7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 76cdda25f78f4f8dab4b5e8cdf63a2b7; no voters:
I20260924 09:58:58.148653 14840 leader_election.cc:288] T 00000000000000000000000000000000 P 76cdda25f78f4f8dab4b5e8cdf63a2b7 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 09:58:58.148804 14844 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 76cdda25f78f4f8dab4b5e8cdf63a2b7 [term 2 FOLLOWER]: Leader election won for term 2
I20260924 09:58:58.149890 14844 raft_consensus.cc:686] T 00000000000000000000000000000000 P 76cdda25f78f4f8dab4b5e8cdf63a2b7 [term 2 LEADER]: Becoming Leader. State: Replica: 76cdda25f78f4f8dab4b5e8cdf63a2b7, State: Running, Role: LEADER
I20260924 09:58:58.150416 14844 consensus_queue.cc:237] T 00000000000000000000000000000000 P 76cdda25f78f4f8dab4b5e8cdf63a2b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76cdda25f78f4f8dab4b5e8cdf63a2b7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46551 } }
I20260924 09:58:58.150825 14840 sys_catalog.cc:564] T 00000000000000000000000000000000 P 76cdda25f78f4f8dab4b5e8cdf63a2b7 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 09:58:58.152841 14845 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76cdda25f78f4f8dab4b5e8cdf63a2b7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "76cdda25f78f4f8dab4b5e8cdf63a2b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76cdda25f78f4f8dab4b5e8cdf63a2b7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46551 } } }
I20260924 09:58:58.153232 14845 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76cdda25f78f4f8dab4b5e8cdf63a2b7 [sys.catalog]: This master's current role is: LEADER
I20260924 09:58:58.156939 14846 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76cdda25f78f4f8dab4b5e8cdf63a2b7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 76cdda25f78f4f8dab4b5e8cdf63a2b7. Latest consensus state: current_term: 2 leader_uuid: "76cdda25f78f4f8dab4b5e8cdf63a2b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76cdda25f78f4f8dab4b5e8cdf63a2b7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46551 } } }
I20260924 09:58:58.157251 14846 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76cdda25f78f4f8dab4b5e8cdf63a2b7 [sys.catalog]: This master's current role is: LEADER
I20260924 09:58:58.158823 14850 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 09:58:58.168394 14850 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 09:58:58.169932 14850 catalog_manager.cc:1259] Loaded cluster ID: e58d63fbf0b544da81d1e5cd9e6c8806
I20260924 09:58:58.170079 14850 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 09:58:58.173333 14850 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260924 09:58:58.189985 14850 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260924 09:58:58.198861 14850 catalog_manager.cc:1504] Loading token signing keys...
I20260924 09:58:58.200664 14850 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 76cdda25f78f4f8dab4b5e8cdf63a2b7: Loaded TSK: 0
I20260924 09:58:58.202317 14850 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 09:58:58.427100 14842 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 09:58:58.427426 14842 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 09:58:58.427578 14842 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 09:58:58.435101 14842 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 09:58:58.435214 14842 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 09:58:58.435381 14842 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260924 09:58:58.445546 14842 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:36377
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=44965
--tserver_master_addrs=127.0.97.62:46551
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 09:58:58.446846 14842 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 09:58:58.448161 14842 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 09:58:58.462590 14867 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:58:58.462581 14868 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:58:58.463205 14870 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 09:58:58.463742 14842 server_base.cc:1034] running on GCE node
I20260924 09:58:58.464313 14842 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 09:58:58.465255 14842 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 09:58:58.466464 14842 hybrid_clock.cc:648] HybridClock initialized: now 1790243938466415 us; error 41 us; skew 500 ppm
I20260924 09:58:58.466756 14842 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 09:58:58 dist-test-slave-w8d2 krb5kdc[14221](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243938, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 09:58:58.469794 14842 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260924 09:58:58.471225 14842 webserver.cc:460] Webserver started at http://127.0.97.1:44965/ using document root <none> and password file <none>
I20260924 09:58:58.471899 14842 fs_manager.cc:359] Metadata directory not provided
I20260924 09:58:58.472023 14842 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 09:58:58.477109 14842 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260924 09:58:58.479739 14877 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 09:58:58.481101 14842 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260924 09:58:58.481245 14842 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/minicluster-data/ts-0/wal
uuid: "fc45f069420445869c886224da489dc6"
format_stamp: "Formatted at 2026-09-24 09:58:55 on dist-test-slave-w8d2"
server_key: "49da492ccbc870cadd378684801b7529"
server_key_iv: "82c3f8a4747d352eaf96db4cf43ace10"
server_key_version: "encryptionkey@0"
I20260924 09:58:58.482210 14842 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 09:58:58.498003 14842 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 09:58:58.502039 14842 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 09:58:58.502411 14842 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 09:58:58.503486 14842 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 09:58:58.504814 14842 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 09:58:58.504905 14842 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 09:58:58.505013 14842 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 09:58:58.505082 14842 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 09:58:58.534689 14842 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:36377
I20260924 09:58:58.534746 14989 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:36377 every 8 connection(s)
I20260924 09:58:58.536092 14842 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/minicluster-data/ts-0/data/info.pb
I20260924 09:58:58.545980 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu as pid 14842
I20260924 09:58:58.546700 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
/tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:40223
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:46551
--webserver_port=44365
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 09:58:58 dist-test-slave-w8d2 krb5kdc[14221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243938, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 09:58:58.559136 14993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:58:58.538868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46551 (local address 127.0.97.1:37081)
0924 09:58:58.539561 (+ 693us) negotiation.cc:107] Waiting for socket to connect
0924 09:58:58.539602 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 09:58:58.541209 (+ 1607us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:58:58.542547 (+ 1338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:58:58.542558 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:58:58.543117 (+ 559us) client_negotiation.cc:189] Negotiated authn=SASL
0924 09:58:58.543709 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:58:58.543727 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:58:58.545331 (+ 1604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:58:58.545335 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:58:58.547184 (+ 1849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:58:58.547200 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:58:58.548199 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:58:58.548206 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:58:58.548454 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:58:58.549134 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 09:58:58.549161 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0924 09:58:58.551797 (+ 2636us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 09:58:58.555416 (+ 3619us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 09:58:58.555423 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 09:58:58.555434 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0924 09:58:58.555875 (+ 441us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 09:58:58.556302 (+ 427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 09:58:58.556305 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 09:58:58.556307 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 09:58:58.556475 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 09:58:58.557243 (+ 768us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 09:58:58.557248 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 09:58:58.557538 (+ 290us) client_negotiation.cc:769] Sending connection context
0924 09:58:58.557743 (+ 205us) client_negotiation.cc:240] Negotiation successful
0924 09:58:58.558127 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":293,"threads_started":1}
I20260924 09:58:58.559966 14992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:58:58.538637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:37081 (local address 127.0.97.62:46551)
0924 09:58:58.539180 (+ 543us) server_negotiation.cc:206] Beginning negotiation
0924 09:58:58.539197 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0924 09:58:58.539732 (+ 535us) server_negotiation.cc:407] Connection header received
0924 09:58:58.541723 (+ 1991us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:58:58.541736 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:58:58.542065 (+ 329us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:58:58.542383 (+ 318us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:58:58.543915 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:58:58.545170 (+ 1255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:58:58.547340 (+ 2170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:58:58.547995 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:58:58.548128 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:58:58.552053 (+ 3925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:58:58.552060 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:58:58.552076 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:58:58.552121 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:58:58.555270 (+ 3149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:58:58.556007 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:58:58.556014 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:58:58.556022 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:58:58.556213 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:58:58.556572 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:58:58.556576 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:58:58.556579 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:58:58.557136 (+ 557us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:58:58.557323 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0924 09:58:58.557973 (+ 650us) server_negotiation.cc:299] Negotiation successful
0924 09:58:58.558362 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338,"thread_start_us":225,"threads_started":1}
I20260924 09:58:58.562572 14990 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46551
I20260924 09:58:58.562845 14990 heartbeater.cc:464] Registering TS with master...
I20260924 09:58:58.563577 14990 heartbeater.cc:511] Master 127.0.97.62:46551 requested a full tablet report, sending...
I20260924 09:58:58.565627 14805 ts_manager.cc:194] Registered new tserver with Master: fc45f069420445869c886224da489dc6 (127.0.97.1:36377)
I20260924 09:58:58.567086 14805 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:37081
W20260924 09:58:58.855571 14994 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 09:58:58.855902 14994 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 09:58:58.856029 14994 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 09:58:58.862964 14994 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 09:58:58.863054 14994 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 09:58:58.863159 14994 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260924 09:58:58.872359 14994 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:40223
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=44365
--tserver_master_addrs=127.0.97.62:46551
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 09:58:58.873468 14994 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 09:58:58.874625 14994 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 09:58:58.886550 14999 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:58:58.886487 15002 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:58:58.886526 15000 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 09:58:58.887446 14994 server_base.cc:1034] running on GCE node
I20260924 09:58:58.888007 14994 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 09:58:58.888885 14994 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 09:58:58.890076 14994 hybrid_clock.cc:648] HybridClock initialized: now 1790243938890016 us; error 56 us; skew 500 ppm
I20260924 09:58:58.890352 14994 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 09:58:58 dist-test-slave-w8d2 krb5kdc[14221](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243938, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 09:58:58.893152 14994 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260924 09:58:58.894454 14994 webserver.cc:460] Webserver started at http://127.0.97.2:44365/ using document root <none> and password file <none>
I20260924 09:58:58.895040 14994 fs_manager.cc:359] Metadata directory not provided
I20260924 09:58:58.895153 14994 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 09:58:58.899996 14994 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260924 09:58:58.902556 15009 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 09:58:58.903766 14994 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260924 09:58:58.903905 14994 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/minicluster-data/ts-1/wal
uuid: "d540747f7bab4b0aa074cda755e1642d"
format_stamp: "Formatted at 2026-09-24 09:58:56 on dist-test-slave-w8d2"
server_key: "0fa54e7f8d9fcddc5e832038d95a6d98"
server_key_iv: "3bb8091213b5eacbce3be064ffdb49ec"
server_key_version: "encryptionkey@0"
I20260924 09:58:58.904786 14994 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 09:58:58.917088 14994 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 09:58:58.920504 14994 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 09:58:58.920737 14994 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 09:58:58.921571 14994 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 09:58:58.922730 14994 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 09:58:58.922818 14994 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 09:58:58.922916 14994 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 09:58:58.922989 14994 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 09:58:58.950031 14994 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:40223
I20260924 09:58:58.950098 15121 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:40223 every 8 connection(s)
I20260924 09:58:58.951428 14994 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/minicluster-data/ts-1/data/info.pb
I20260924 09:58:58.959856 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu as pid 14994
I20260924 09:58:58.960421 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
/tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:39885
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:46551
--webserver_port=41853
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 09:58:58 dist-test-slave-w8d2 krb5kdc[14221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243938, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 09:58:58.969331 14992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:58:58.953779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:45525 (local address 127.0.97.62:46551)
0924 09:58:58.953938 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0924 09:58:58.953942 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 09:58:58.955039 (+ 1097us) server_negotiation.cc:407] Connection header received
0924 09:58:58.956716 (+ 1677us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:58:58.956720 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:58:58.956788 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:58:58.956872 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:58:58.958180 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:58:58.958874 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:58:58.960431 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:58:58.960921 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:58:58.961032 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:58:58.964340 (+ 3308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:58:58.964345 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:58:58.964348 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:58:58.964379 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:58:58.966605 (+ 2226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:58:58.967297 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:58:58.967302 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:58:58.967304 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:58:58.967429 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:58:58.967847 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:58:58.967850 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:58:58.967852 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:58:58.968107 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:58:58.968214 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0924 09:58:58.968918 (+ 704us) server_negotiation.cc:299] Negotiation successful
0924 09:58:58.969094 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 09:58:58.970559 15124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:58:58.954233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46551 (local address 127.0.97.2:45525)
0924 09:58:58.954894 (+ 661us) negotiation.cc:107] Waiting for socket to connect
0924 09:58:58.954929 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 09:58:58.956490 (+ 1561us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:58:58.957047 (+ 557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:58:58.957055 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:58:58.957508 (+ 453us) client_negotiation.cc:189] Negotiated authn=SASL
0924 09:58:58.958034 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:58:58.958050 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:58:58.958995 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:58:58.958999 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:58:58.960301 (+ 1302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:58:58.960312 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:58:58.961126 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:58:58.961133 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:58:58.961353 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:58:58.961964 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 09:58:58.961984 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 09:58:58.964145 (+ 2161us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 09:58:58.966742 (+ 2597us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 09:58:58.966750 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 09:58:58.966764 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 09:58:58.967147 (+ 383us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 09:58:58.967521 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 09:58:58.967526 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 09:58:58.967530 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 09:58:58.967760 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 09:58:58.968208 (+ 448us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 09:58:58.968215 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 09:58:58.968568 (+ 353us) client_negotiation.cc:769] Sending connection context
0924 09:58:58.968842 (+ 274us) client_negotiation.cc:240] Negotiation successful
0924 09:58:58.969325 (+ 483us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":258,"threads_started":1}
I20260924 09:58:58.971980 15122 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46551
I20260924 09:58:58.972292 15122 heartbeater.cc:464] Registering TS with master...
I20260924 09:58:58.973045 15122 heartbeater.cc:511] Master 127.0.97.62:46551 requested a full tablet report, sending...
I20260924 09:58:58.974422 14805 ts_manager.cc:194] Registered new tserver with Master: d540747f7bab4b0aa074cda755e1642d (127.0.97.2:40223)
I20260924 09:58:58.975193 14805 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:45525
W20260924 09:58:59.264398 15125 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 09:58:59.264664 15125 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 09:58:59.264796 15125 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 09:58:59.271950 15125 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 09:58:59.272060 15125 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 09:58:59.272202 15125 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260924 09:58:59.281541 15125 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:39885
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=41853
--tserver_master_addrs=127.0.97.62:46551
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 09:58:59.282811 15125 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 09:58:59.284080 15125 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 09:58:59.296545 15131 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:58:59.296665 15130 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:58:59.296631 15133 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 09:58:59.297214 15125 server_base.cc:1034] running on GCE node
I20260924 09:58:59.297758 15125 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 09:58:59.298583 15125 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 09:58:59.299765 15125 hybrid_clock.cc:648] HybridClock initialized: now 1790243939299720 us; error 38 us; skew 500 ppm
I20260924 09:58:59.300028 15125 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 09:58:59 dist-test-slave-w8d2 krb5kdc[14221](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243939, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 09:58:59.302801 15125 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260924 09:58:59.304150 15125 webserver.cc:460] Webserver started at http://127.0.97.3:41853/ using document root <none> and password file <none>
I20260924 09:58:59.304771 15125 fs_manager.cc:359] Metadata directory not provided
I20260924 09:58:59.304894 15125 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 09:58:59.309947 15125 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20260924 09:58:59.312453 15140 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 09:58:59.313652 15125 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 09:58:59.313791 15125 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/minicluster-data/ts-2/wal
uuid: "df6733a6ac4a4b7b87437920eaa15585"
format_stamp: "Formatted at 2026-09-24 09:58:56 on dist-test-slave-w8d2"
server_key: "eb52ee37941673b737f37b0fa0e5abe5"
server_key_iv: "2116980e992b0b4705d1395bb478eccc"
server_key_version: "encryptionkey@0"
I20260924 09:58:59.314697 15125 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 09:58:59.337214 15125 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 09:58:59.341325 15125 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 09:58:59.341590 15125 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 09:58:59.342506 15125 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 09:58:59.343833 15125 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 09:58:59.343919 15125 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 09:58:59.344003 15125 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 09:58:59.344055 15125 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 09:58:59.373333 15125 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:39885
I20260924 09:58:59.373396 15252 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:39885 every 8 connection(s)
I20260924 09:58:59.374694 15125 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/minicluster-data/ts-2/data/info.pb
I20260924 09:58:59.382433 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu as pid 15125
Sep 24 09:58:59 dist-test-slave-w8d2 krb5kdc[14221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243939, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 09:58:59.393718 14992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:58:59.377113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:50127 (local address 127.0.97.62:46551)
0924 09:58:59.377294 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0924 09:58:59.377298 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 09:58:59.378416 (+ 1118us) server_negotiation.cc:407] Connection header received
0924 09:58:59.380152 (+ 1736us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:58:59.380157 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:58:59.380222 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:58:59.380311 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:58:59.381641 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:58:59.382511 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:58:59.384396 (+ 1885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:58:59.384817 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:58:59.384925 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:58:59.388167 (+ 3242us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:58:59.388173 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:58:59.388178 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:58:59.388216 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:58:59.390741 (+ 2525us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:58:59.391522 (+ 781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:58:59.391527 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:58:59.391530 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:58:59.391682 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:58:59.392081 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:58:59.392085 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:58:59.392087 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:58:59.392368 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:58:59.392504 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0924 09:58:59.393161 (+ 657us) server_negotiation.cc:299] Negotiation successful
0924 09:58:59.393394 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260924 09:58:59.394577 15255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:58:59.377576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46551 (local address 127.0.97.3:50127)
0924 09:58:59.378259 (+ 683us) negotiation.cc:107] Waiting for socket to connect
0924 09:58:59.378295 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 09:58:59.379931 (+ 1636us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:58:59.380497 (+ 566us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:58:59.380504 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:58:59.380974 (+ 470us) client_negotiation.cc:189] Negotiated authn=SASL
0924 09:58:59.381514 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:58:59.381531 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:58:59.382653 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:58:59.382656 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:58:59.384259 (+ 1603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:58:59.384271 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:58:59.385049 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:58:59.385057 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:58:59.385290 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:58:59.385868 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 09:58:59.385895 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0924 09:58:59.387981 (+ 2086us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 09:58:59.390878 (+ 2897us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 09:58:59.390886 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 09:58:59.390900 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 09:58:59.391408 (+ 508us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 09:58:59.391785 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 09:58:59.391791 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 09:58:59.391795 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 09:58:59.391966 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 09:58:59.392491 (+ 525us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 09:58:59.392498 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 09:58:59.392798 (+ 300us) client_negotiation.cc:769] Sending connection context
0924 09:58:59.393014 (+ 216us) client_negotiation.cc:240] Negotiation successful
0924 09:58:59.393483 (+ 469us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":272,"threads_started":1}
I20260924 09:58:59.395951 15253 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46551
I20260924 09:58:59.396229 15256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:58:59.383293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47744 (local address 127.0.97.62:46551)
0924 09:58:59.383749 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0924 09:58:59.383756 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 09:58:59.384032 (+ 276us) server_negotiation.cc:407] Connection header received
0924 09:58:59.384178 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:58:59.384181 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:58:59.384246 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:58:59.384324 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:58:59.385795 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:58:59.386564 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:58:59.387886 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:58:59.388452 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:58:59.388560 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:58:59.391122 (+ 2562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:58:59.391126 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:58:59.391129 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:58:59.391155 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:58:59.393679 (+ 2524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:58:59.394412 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:58:59.394417 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:58:59.394420 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:58:59.394525 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:58:59.394888 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:58:59.394893 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:58:59.394896 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:58:59.395132 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:58:59.395371 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0924 09:58:59.395647 (+ 276us) server_negotiation.cc:299] Negotiation successful
0924 09:58:59.395846 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":346,"thread_start_us":231,"threads_started":1}
I20260924 09:58:59.396246 15253 heartbeater.cc:464] Registering TS with master...
I20260924 09:58:59.397100 15253 heartbeater.cc:511] Master 127.0.97.62:46551 requested a full tablet report, sending...
I20260924 09:58:59.398470 14805 ts_manager.cc:194] Registered new tserver with Master: df6733a6ac4a4b7b87437920eaa15585 (127.0.97.3:39885)
I20260924 09:58:59.399263 14805 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:50127
I20260924 09:58:59.410359 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 09:58:59.416282 15256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:58:59.411146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47756 (local address 127.0.97.62:46551)
0924 09:58:59.411335 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0924 09:58:59.411338 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 09:58:59.411681 (+ 343us) server_negotiation.cc:407] Connection header received
0924 09:58:59.411851 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:58:59.411855 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:58:59.411984 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:58:59.412098 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 09:58:59.412394 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:58:59.413049 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:58:59.414391 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:58:59.414815 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:58:59.414905 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:58:59.415345 (+ 440us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 09:58:59.415839 (+ 494us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 09:58:59.415948 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0924 09:58:59.415987 (+ 39us) server_negotiation.cc:299] Negotiation successful
0924 09:58:59.416047 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 09:58:59.417052 14805 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:47756:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 09:58:59.424089 14805 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 09:58:59.458463 15265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:58:59.435868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60162 (local address 127.0.97.3:39885)
0924 09:58:59.451142 (+ 15274us) server_negotiation.cc:206] Beginning negotiation
0924 09:58:59.451150 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 09:58:59.451178 (+ 28us) server_negotiation.cc:407] Connection header received
0924 09:58:59.451306 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:58:59.451315 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:58:59.451534 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:58:59.451940 (+ 406us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 09:58:59.452515 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:58:59.453946 (+ 1431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:58:59.456128 (+ 2182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:58:59.457279 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:58:59.457388 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:58:59.457604 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0924 09:58:59.458001 (+ 397us) server_negotiation.cc:299] Negotiation successful
0924 09:58:59.458153 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15095,"thread_start_us":15807,"threads_started":1}
I20260924 09:58:59.458521 15267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:58:59.436572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:39885 (local address 127.0.0.1:60162)
0924 09:58:59.450289 (+ 13717us) negotiation.cc:107] Waiting for socket to connect
0924 09:58:59.450306 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 09:58:59.450410 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:58:59.451952 (+ 1542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:58:59.451957 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:58:59.451984 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 09:58:59.452330 (+ 346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:58:59.452340 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:58:59.454131 (+ 1791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:58:59.454140 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:58:59.455975 (+ 1835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:58:59.455989 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:58:59.457532 (+ 1543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:58:59.457538 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:58:59.457836 (+ 298us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:58:59.457861 (+ 25us) client_negotiation.cc:769] Sending connection context
0924 09:58:59.457946 (+ 85us) client_negotiation.cc:240] Negotiation successful
0924 09:58:59.458135 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13600,"mutex_wait_us":142,"thread_start_us":12889,"threads_started":1}
I20260924 09:58:59.458796 15266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:58:59.436036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36878 (local address 127.0.97.2:40223)
0924 09:58:59.450911 (+ 14875us) server_negotiation.cc:206] Beginning negotiation
0924 09:58:59.450920 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 09:58:59.450953 (+ 33us) server_negotiation.cc:407] Connection header received
0924 09:58:59.451086 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:58:59.451093 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:58:59.451377 (+ 284us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:58:59.451612 (+ 235us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 09:58:59.452132 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:58:59.453539 (+ 1407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:58:59.455692 (+ 2153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:58:59.456988 (+ 1296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:58:59.457110 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:58:59.457319 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0924 09:58:59.458308 (+ 989us) server_negotiation.cc:299] Negotiation successful
0924 09:58:59.458456 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14700,"thread_start_us":14901,"threads_started":1}
I20260924 09:58:59.459066 15263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:58:59.435223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44512 (local address 127.0.97.1:36377)
0924 09:58:59.447805 (+ 12582us) server_negotiation.cc:206] Beginning negotiation
0924 09:58:59.447817 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 09:58:59.449136 (+ 1319us) server_negotiation.cc:407] Connection header received
0924 09:58:59.449399 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:58:59.449409 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:58:59.449686 (+ 277us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:58:59.449872 (+ 186us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 09:58:59.450844 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:58:59.452436 (+ 1592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:58:59.454998 (+ 2562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:58:59.456251 (+ 1253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:58:59.456376 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:58:59.456574 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0924 09:58:59.458079 (+ 1505us) server_negotiation.cc:299] Negotiation successful
0924 09:58:59.458259 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12359,"thread_start_us":12593,"threads_started":1}
I20260924 09:58:59.459345 15268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:58:59.435499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:40223 (local address 127.0.0.1:36878)
0924 09:58:59.449710 (+ 14211us) negotiation.cc:107] Waiting for socket to connect
0924 09:58:59.449732 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 09:58:59.449840 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:58:59.451567 (+ 1727us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:58:59.451572 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:58:59.451595 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 09:58:59.451945 (+ 350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:58:59.451959 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:58:59.453707 (+ 1748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:58:59.453713 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:58:59.455532 (+ 1819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:58:59.455546 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:58:59.457485 (+ 1939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:58:59.457492 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:58:59.457764 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:58:59.457835 (+ 71us) client_negotiation.cc:769] Sending connection context
0924 09:58:59.457951 (+ 116us) client_negotiation.cc:240] Negotiation successful
0924 09:58:59.458130 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14084,"mutex_wait_us":14,"thread_start_us":14535,"threads_started":1}
I20260924 09:58:59.459688 15264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:58:59.435210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36377 (local address 127.0.0.1:44512)
0924 09:58:59.448960 (+ 13750us) negotiation.cc:107] Waiting for socket to connect
0924 09:58:59.448998 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 09:58:59.449187 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:58:59.450174 (+ 987us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:58:59.450182 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:58:59.450223 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 09:58:59.450639 (+ 416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:58:59.450653 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:58:59.452530 (+ 1877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:58:59.452535 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:58:59.454821 (+ 2286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:58:59.454836 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:58:59.457472 (+ 2636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:58:59.457480 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:58:59.457773 (+ 293us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:58:59.457829 (+ 56us) client_negotiation.cc:769] Sending connection context
0924 09:58:59.457951 (+ 122us) client_negotiation.cc:240] Negotiation successful
0924 09:58:59.458780 (+ 829us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13143,"mutex_wait_us":755,"thread_start_us":12976,"threads_started":1}
I20260924 09:58:59.462816 15057 tablet_service.cc:1467] Processing CreateTablet for tablet e4b9ffaa1adf4096995644923cd427d1 (DEFAULT_TABLE table=test-table [id=73006212c0a148f596f01ad14d5d3bf8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 09:58:59.462873 15188 tablet_service.cc:1467] Processing CreateTablet for tablet e4b9ffaa1adf4096995644923cd427d1 (DEFAULT_TABLE table=test-table [id=73006212c0a148f596f01ad14d5d3bf8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 09:58:59.462963 14925 tablet_service.cc:1467] Processing CreateTablet for tablet e4b9ffaa1adf4096995644923cd427d1 (DEFAULT_TABLE table=test-table [id=73006212c0a148f596f01ad14d5d3bf8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 09:58:59.464061 15057 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4b9ffaa1adf4096995644923cd427d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 09:58:59.464203 15188 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4b9ffaa1adf4096995644923cd427d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 09:58:59.464205 14925 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4b9ffaa1adf4096995644923cd427d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 09:58:59.475378 15269 tablet_bootstrap.cc:492] T e4b9ffaa1adf4096995644923cd427d1 P d540747f7bab4b0aa074cda755e1642d: Bootstrap starting.
I20260924 09:58:59.478663 15270 tablet_bootstrap.cc:492] T e4b9ffaa1adf4096995644923cd427d1 P fc45f069420445869c886224da489dc6: Bootstrap starting.
I20260924 09:58:59.479564 15269 tablet_bootstrap.cc:654] T e4b9ffaa1adf4096995644923cd427d1 P d540747f7bab4b0aa074cda755e1642d: Neither blocks nor log segments found. Creating new log.
I20260924 09:58:59.480791 15269 log.cc:824] T e4b9ffaa1adf4096995644923cd427d1 P d540747f7bab4b0aa074cda755e1642d: Log is configured to *not* fsync() on all Append() calls
I20260924 09:58:59.482033 15270 tablet_bootstrap.cc:654] T e4b9ffaa1adf4096995644923cd427d1 P fc45f069420445869c886224da489dc6: Neither blocks nor log segments found. Creating new log.
I20260924 09:58:59.483198 15270 log.cc:824] T e4b9ffaa1adf4096995644923cd427d1 P fc45f069420445869c886224da489dc6: Log is configured to *not* fsync() on all Append() calls
I20260924 09:58:59.484026 15269 tablet_bootstrap.cc:492] T e4b9ffaa1adf4096995644923cd427d1 P d540747f7bab4b0aa074cda755e1642d: No bootstrap required, opened a new log
I20260924 09:58:59.484017 15271 tablet_bootstrap.cc:492] T e4b9ffaa1adf4096995644923cd427d1 P df6733a6ac4a4b7b87437920eaa15585: Bootstrap starting.
I20260924 09:58:59.484268 15269 ts_tablet_manager.cc:1397] T e4b9ffaa1adf4096995644923cd427d1 P d540747f7bab4b0aa074cda755e1642d: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260924 09:58:59.486141 15270 tablet_bootstrap.cc:492] T e4b9ffaa1adf4096995644923cd427d1 P fc45f069420445869c886224da489dc6: No bootstrap required, opened a new log
I20260924 09:58:59.486380 15270 ts_tablet_manager.cc:1397] T e4b9ffaa1adf4096995644923cd427d1 P fc45f069420445869c886224da489dc6: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260924 09:58:59.486786 15271 tablet_bootstrap.cc:654] T e4b9ffaa1adf4096995644923cd427d1 P df6733a6ac4a4b7b87437920eaa15585: Neither blocks nor log segments found. Creating new log.
I20260924 09:58:59.487839 15271 log.cc:824] T e4b9ffaa1adf4096995644923cd427d1 P df6733a6ac4a4b7b87437920eaa15585: Log is configured to *not* fsync() on all Append() calls
I20260924 09:58:59.490113 15271 tablet_bootstrap.cc:492] T e4b9ffaa1adf4096995644923cd427d1 P df6733a6ac4a4b7b87437920eaa15585: No bootstrap required, opened a new log
I20260924 09:58:59.490331 15271 ts_tablet_manager.cc:1397] T e4b9ffaa1adf4096995644923cd427d1 P df6733a6ac4a4b7b87437920eaa15585: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260924 09:58:59.494769 15269 raft_consensus.cc:348] T e4b9ffaa1adf4096995644923cd427d1 P d540747f7bab4b0aa074cda755e1642d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc45f069420445869c886224da489dc6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36377 } } peers { permanent_uuid: "d540747f7bab4b0aa074cda755e1642d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40223 } } peers { permanent_uuid: "df6733a6ac4a4b7b87437920eaa15585" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39885 } }
I20260924 09:58:59.495175 15269 raft_consensus.cc:374] T e4b9ffaa1adf4096995644923cd427d1 P d540747f7bab4b0aa074cda755e1642d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 09:58:59.495297 15269 raft_consensus.cc:729] T e4b9ffaa1adf4096995644923cd427d1 P d540747f7bab4b0aa074cda755e1642d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d540747f7bab4b0aa074cda755e1642d, State: Initialized, Role: FOLLOWER
I20260924 09:58:59.495890 15269 consensus_queue.cc:260] T e4b9ffaa1adf4096995644923cd427d1 P d540747f7bab4b0aa074cda755e1642d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc45f069420445869c886224da489dc6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36377 } } peers { permanent_uuid: "d540747f7bab4b0aa074cda755e1642d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40223 } } peers { permanent_uuid: "df6733a6ac4a4b7b87437920eaa15585" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39885 } }
I20260924 09:58:59.496297 15270 raft_consensus.cc:348] T e4b9ffaa1adf4096995644923cd427d1 P fc45f069420445869c886224da489dc6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc45f069420445869c886224da489dc6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36377 } } peers { permanent_uuid: "d540747f7bab4b0aa074cda755e1642d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40223 } } peers { permanent_uuid: "df6733a6ac4a4b7b87437920eaa15585" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39885 } }
I20260924 09:58:59.496713 15270 raft_consensus.cc:374] T e4b9ffaa1adf4096995644923cd427d1 P fc45f069420445869c886224da489dc6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 09:58:59.496806 15270 raft_consensus.cc:729] T e4b9ffaa1adf4096995644923cd427d1 P fc45f069420445869c886224da489dc6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc45f069420445869c886224da489dc6, State: Initialized, Role: FOLLOWER
I20260924 09:58:59.497116 15122 heartbeater.cc:503] Master 127.0.97.62:46551 was elected leader, sending a full tablet report...
I20260924 09:58:59.497412 15270 consensus_queue.cc:260] T e4b9ffaa1adf4096995644923cd427d1 P fc45f069420445869c886224da489dc6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc45f069420445869c886224da489dc6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36377 } } peers { permanent_uuid: "d540747f7bab4b0aa074cda755e1642d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40223 } } peers { permanent_uuid: "df6733a6ac4a4b7b87437920eaa15585" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39885 } }
I20260924 09:58:59.497852 15269 ts_tablet_manager.cc:1428] T e4b9ffaa1adf4096995644923cd427d1 P d540747f7bab4b0aa074cda755e1642d: Time spent starting tablet: real 0.013s user 0.005s sys 0.008s
I20260924 09:58:59.497675 15271 raft_consensus.cc:348] T e4b9ffaa1adf4096995644923cd427d1 P df6733a6ac4a4b7b87437920eaa15585 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc45f069420445869c886224da489dc6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36377 } } peers { permanent_uuid: "d540747f7bab4b0aa074cda755e1642d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40223 } } peers { permanent_uuid: "df6733a6ac4a4b7b87437920eaa15585" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39885 } }
I20260924 09:58:59.498031 15271 raft_consensus.cc:374] T e4b9ffaa1adf4096995644923cd427d1 P df6733a6ac4a4b7b87437920eaa15585 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 09:58:59.498118 15271 raft_consensus.cc:729] T e4b9ffaa1adf4096995644923cd427d1 P df6733a6ac4a4b7b87437920eaa15585 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df6733a6ac4a4b7b87437920eaa15585, State: Initialized, Role: FOLLOWER
I20260924 09:58:59.498705 14990 heartbeater.cc:503] Master 127.0.97.62:46551 was elected leader, sending a full tablet report...
I20260924 09:58:59.498709 15271 consensus_queue.cc:260] T e4b9ffaa1adf4096995644923cd427d1 P df6733a6ac4a4b7b87437920eaa15585 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc45f069420445869c886224da489dc6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36377 } } peers { permanent_uuid: "d540747f7bab4b0aa074cda755e1642d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40223 } } peers { permanent_uuid: "df6733a6ac4a4b7b87437920eaa15585" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39885 } }
I20260924 09:58:59.500643 15270 ts_tablet_manager.cc:1428] T e4b9ffaa1adf4096995644923cd427d1 P fc45f069420445869c886224da489dc6: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
I20260924 09:58:59.501508 15253 heartbeater.cc:503] Master 127.0.97.62:46551 was elected leader, sending a full tablet report...
I20260924 09:58:59.502979 15271 ts_tablet_manager.cc:1428] T e4b9ffaa1adf4096995644923cd427d1 P df6733a6ac4a4b7b87437920eaa15585: Time spent starting tablet: real 0.013s user 0.008s sys 0.005s
W20260924 09:58:59.538067 14991 tablet.cc:2369] T e4b9ffaa1adf4096995644923cd427d1 P fc45f069420445869c886224da489dc6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 09:58:59.604347 15277 raft_consensus.cc:482] T e4b9ffaa1adf4096995644923cd427d1 P df6733a6ac4a4b7b87437920eaa15585 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 09:58:59.604633 15277 raft_consensus.cc:504] T e4b9ffaa1adf4096995644923cd427d1 P df6733a6ac4a4b7b87437920eaa15585 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc45f069420445869c886224da489dc6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36377 } } peers { permanent_uuid: "d540747f7bab4b0aa074cda755e1642d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40223 } } peers { permanent_uuid: "df6733a6ac4a4b7b87437920eaa15585" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39885 } }
I20260924 09:58:59.606308 15277 leader_election.cc:288] T e4b9ffaa1adf4096995644923cd427d1 P df6733a6ac4a4b7b87437920eaa15585 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fc45f069420445869c886224da489dc6 (127.0.97.1:36377), d540747f7bab4b0aa074cda755e1642d (127.0.97.2:40223)
I20260924 09:58:59.612197 15263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:58:59.606877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:47245 (local address 127.0.97.1:36377)
0924 09:58:59.607069 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0924 09:58:59.607075 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 09:58:59.607092 (+ 17us) server_negotiation.cc:407] Connection header received
0924 09:58:59.607169 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:58:59.607174 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:58:59.607281 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:58:59.607372 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 09:58:59.607774 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:58:59.608662 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:58:59.610371 (+ 1709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:58:59.611366 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:58:59.611462 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:58:59.611530 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 09:58:59.611837 (+ 307us) server_negotiation.cc:299] Negotiation successful
0924 09:58:59.611935 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260924 09:58:59.612205 15255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:58:59.606470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36377 (local address 127.0.97.3:47245)
0924 09:58:59.606704 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0924 09:58:59.606723 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 09:58:59.606848 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:58:59.607402 (+ 554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:58:59.607406 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:58:59.607430 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 09:58:59.607620 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:58:59.607631 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:58:59.608789 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:58:59.608794 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:58:59.610218 (+ 1424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:58:59.610229 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:58:59.611505 (+ 1276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:58:59.611511 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:58:59.611684 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:58:59.611718 (+ 34us) client_negotiation.cc:769] Sending connection context
0924 09:58:59.611823 (+ 105us) client_negotiation.cc:240] Negotiation successful
0924 09:58:59.611943 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":73}
I20260924 09:58:59.612751 15266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:58:59.606796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:46747 (local address 127.0.97.2:40223)
0924 09:58:59.606984 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0924 09:58:59.606990 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 09:58:59.607514 (+ 524us) server_negotiation.cc:407] Connection header received
0924 09:58:59.607657 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:58:59.607662 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:58:59.607735 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:58:59.607805 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 09:58:59.608351 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:58:59.609139 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:58:59.610920 (+ 1781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:58:59.611846 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:58:59.611947 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:58:59.612034 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0924 09:58:59.612397 (+ 363us) server_negotiation.cc:299] Negotiation successful
0924 09:58:59.612524 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260924 09:58:59.613219 15278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:58:59.606717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:40223 (local address 127.0.97.3:46747)
0924 09:58:59.607416 (+ 699us) negotiation.cc:107] Waiting for socket to connect
0924 09:58:59.607433 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 09:58:59.607547 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:58:59.607838 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:58:59.607842 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:58:59.607867 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 09:58:59.608197 (+ 330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:58:59.608209 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:58:59.609264 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:58:59.609271 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:58:59.610777 (+ 1506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:58:59.610791 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:58:59.612024 (+ 1233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:58:59.612030 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:58:59.612215 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:58:59.612247 (+ 32us) client_negotiation.cc:769] Sending connection context
0924 09:58:59.612904 (+ 657us) client_negotiation.cc:240] Negotiation successful
0924 09:58:59.612997 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":567,"thread_start_us":373,"threads_started":1}
I20260924 09:58:59.613191 14945 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4b9ffaa1adf4096995644923cd427d1" candidate_uuid: "df6733a6ac4a4b7b87437920eaa15585" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc45f069420445869c886224da489dc6" is_pre_election: true
I20260924 09:58:59.613704 14945 raft_consensus.cc:2454] T e4b9ffaa1adf4096995644923cd427d1 P fc45f069420445869c886224da489dc6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6733a6ac4a4b7b87437920eaa15585 in term 0.
I20260924 09:58:59.614122 15077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4b9ffaa1adf4096995644923cd427d1" candidate_uuid: "df6733a6ac4a4b7b87437920eaa15585" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d540747f7bab4b0aa074cda755e1642d" is_pre_election: true
I20260924 09:58:59.614362 15144 leader_election.cc:302] T e4b9ffaa1adf4096995644923cd427d1 P df6733a6ac4a4b7b87437920eaa15585 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6733a6ac4a4b7b87437920eaa15585, fc45f069420445869c886224da489dc6; no voters:
I20260924 09:58:59.614602 15077 raft_consensus.cc:2454] T e4b9ffaa1adf4096995644923cd427d1 P d540747f7bab4b0aa074cda755e1642d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6733a6ac4a4b7b87437920eaa15585 in term 0.
I20260924 09:58:59.614740 15277 raft_consensus.cc:2781] T e4b9ffaa1adf4096995644923cd427d1 P df6733a6ac4a4b7b87437920eaa15585 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 09:58:59.614882 15277 raft_consensus.cc:482] T e4b9ffaa1adf4096995644923cd427d1 P df6733a6ac4a4b7b87437920eaa15585 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 09:58:59.614979 15277 raft_consensus.cc:3037] T e4b9ffaa1adf4096995644923cd427d1 P df6733a6ac4a4b7b87437920eaa15585 [term 0 FOLLOWER]: Advancing to term 1
I20260924 09:58:59.617164 15277 raft_consensus.cc:504] T e4b9ffaa1adf4096995644923cd427d1 P df6733a6ac4a4b7b87437920eaa15585 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc45f069420445869c886224da489dc6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36377 } } peers { permanent_uuid: "d540747f7bab4b0aa074cda755e1642d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40223 } } peers { permanent_uuid: "df6733a6ac4a4b7b87437920eaa15585" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39885 } }
I20260924 09:58:59.617663 15277 leader_election.cc:288] T e4b9ffaa1adf4096995644923cd427d1 P df6733a6ac4a4b7b87437920eaa15585 [CANDIDATE]: Term 1 election: Requested vote from peers fc45f069420445869c886224da489dc6 (127.0.97.1:36377), d540747f7bab4b0aa074cda755e1642d (127.0.97.2:40223)
I20260924 09:58:59.618029 14945 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4b9ffaa1adf4096995644923cd427d1" candidate_uuid: "df6733a6ac4a4b7b87437920eaa15585" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc45f069420445869c886224da489dc6"
I20260924 09:58:59.618135 15077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4b9ffaa1adf4096995644923cd427d1" candidate_uuid: "df6733a6ac4a4b7b87437920eaa15585" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d540747f7bab4b0aa074cda755e1642d"
I20260924 09:58:59.618221 14945 raft_consensus.cc:3037] T e4b9ffaa1adf4096995644923cd427d1 P fc45f069420445869c886224da489dc6 [term 0 FOLLOWER]: Advancing to term 1
I20260924 09:58:59.618314 15077 raft_consensus.cc:3037] T e4b9ffaa1adf4096995644923cd427d1 P d540747f7bab4b0aa074cda755e1642d [term 0 FOLLOWER]: Advancing to term 1
I20260924 09:58:59.621274 14945 raft_consensus.cc:2454] T e4b9ffaa1adf4096995644923cd427d1 P fc45f069420445869c886224da489dc6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6733a6ac4a4b7b87437920eaa15585 in term 1.
I20260924 09:58:59.621268 15077 raft_consensus.cc:2454] T e4b9ffaa1adf4096995644923cd427d1 P d540747f7bab4b0aa074cda755e1642d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6733a6ac4a4b7b87437920eaa15585 in term 1.
I20260924 09:58:59.621585 15144 leader_election.cc:302] T e4b9ffaa1adf4096995644923cd427d1 P df6733a6ac4a4b7b87437920eaa15585 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6733a6ac4a4b7b87437920eaa15585, fc45f069420445869c886224da489dc6; no voters:
I20260924 09:58:59.621805 15277 raft_consensus.cc:2781] T e4b9ffaa1adf4096995644923cd427d1 P df6733a6ac4a4b7b87437920eaa15585 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 09:58:59.622195 15277 raft_consensus.cc:686] T e4b9ffaa1adf4096995644923cd427d1 P df6733a6ac4a4b7b87437920eaa15585 [term 1 LEADER]: Becoming Leader. State: Replica: df6733a6ac4a4b7b87437920eaa15585, State: Running, Role: LEADER
I20260924 09:58:59.622680 15277 consensus_queue.cc:237] T e4b9ffaa1adf4096995644923cd427d1 P df6733a6ac4a4b7b87437920eaa15585 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc45f069420445869c886224da489dc6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36377 } } peers { permanent_uuid: "d540747f7bab4b0aa074cda755e1642d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40223 } } peers { permanent_uuid: "df6733a6ac4a4b7b87437920eaa15585" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39885 } }
W20260924 09:58:59.626087 15254 tablet.cc:2369] T e4b9ffaa1adf4096995644923cd427d1 P df6733a6ac4a4b7b87437920eaa15585: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 09:58:59.626950 14804 catalog_manager.cc:5495] T e4b9ffaa1adf4096995644923cd427d1 P df6733a6ac4a4b7b87437920eaa15585 reported cstate change: term changed from 0 to 1, leader changed from <none> to df6733a6ac4a4b7b87437920eaa15585 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "df6733a6ac4a4b7b87437920eaa15585" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc45f069420445869c886224da489dc6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36377 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d540747f7bab4b0aa074cda755e1642d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40223 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df6733a6ac4a4b7b87437920eaa15585" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39885 } health_report { overall_health: HEALTHY } } }
I20260924 09:58:59.648994 15265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:58:59.644106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60168 (local address 127.0.97.3:39885)
0924 09:58:59.644292 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0924 09:58:59.644297 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 09:58:59.644315 (+ 18us) server_negotiation.cc:407] Connection header received
0924 09:58:59.644472 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:58:59.644476 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:58:59.644546 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:58:59.644661 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 09:58:59.645031 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:58:59.645719 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:58:59.647004 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:58:59.647436 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:58:59.647542 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:58:59.647992 (+ 450us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 09:58:59.648459 (+ 467us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 09:58:59.648593 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0924 09:58:59.648677 (+ 84us) server_negotiation.cc:299] Negotiation successful
0924 09:58:59.648753 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 09:58:59.656737 14945 raft_consensus.cc:1261] T e4b9ffaa1adf4096995644923cd427d1 P fc45f069420445869c886224da489dc6 [term 1 FOLLOWER]: Refusing update from remote peer df6733a6ac4a4b7b87437920eaa15585: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 09:58:59.656738 15077 raft_consensus.cc:1261] T e4b9ffaa1adf4096995644923cd427d1 P d540747f7bab4b0aa074cda755e1642d [term 1 FOLLOWER]: Refusing update from remote peer df6733a6ac4a4b7b87437920eaa15585: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 09:58:59.657737 15279 consensus_queue.cc:1035] T e4b9ffaa1adf4096995644923cd427d1 P df6733a6ac4a4b7b87437920eaa15585 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d540747f7bab4b0aa074cda755e1642d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 09:58:59.658048 15277 consensus_queue.cc:1035] T e4b9ffaa1adf4096995644923cd427d1 P df6733a6ac4a4b7b87437920eaa15585 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc45f069420445869c886224da489dc6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 09:58:59.682642 14804 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:47756:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 09:58:59.682881 14804 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:47756:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 09:58:59.685956 14804 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 76cdda25f78f4f8dab4b5e8cdf63a2b7: Sending DeleteTablet for 3 replicas of tablet e4b9ffaa1adf4096995644923cd427d1
I20260924 09:58:59.687077 14925 tablet_service.cc:1514] Processing DeleteTablet for tablet e4b9ffaa1adf4096995644923cd427d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 09:58:59 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:44512
I20260924 09:58:59.687077 15057 tablet_service.cc:1514] Processing DeleteTablet for tablet e4b9ffaa1adf4096995644923cd427d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 09:58:59 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:36878
I20260924 09:58:59.687188 15188 tablet_service.cc:1514] Processing DeleteTablet for tablet e4b9ffaa1adf4096995644923cd427d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 09:58:59 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:60162
I20260924 09:58:59.689677 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu pid 14842
I20260924 09:58:59.698001 15295 tablet_replica.cc:330] T e4b9ffaa1adf4096995644923cd427d1 P df6733a6ac4a4b7b87437920eaa15585: stopping tablet replica
I20260924 09:58:59.698500 15295 raft_consensus.cc:2229] T e4b9ffaa1adf4096995644923cd427d1 P df6733a6ac4a4b7b87437920eaa15585 [term 1 LEADER]: Raft consensus shutting down.
I20260924 09:58:59.702350 15294 tablet_replica.cc:330] T e4b9ffaa1adf4096995644923cd427d1 P fc45f069420445869c886224da489dc6: stopping tablet replica
I20260924 09:58:59.702811 15294 raft_consensus.cc:2229] T e4b9ffaa1adf4096995644923cd427d1 P fc45f069420445869c886224da489dc6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 09:58:59.704253 15295 raft_consensus.cc:2258] T e4b9ffaa1adf4096995644923cd427d1 P df6733a6ac4a4b7b87437920eaa15585 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 09:58:59.705830 15294 raft_consensus.cc:2258] T e4b9ffaa1adf4096995644923cd427d1 P fc45f069420445869c886224da489dc6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 09:58:59.706771 15295 ts_tablet_manager.cc:1905] T e4b9ffaa1adf4096995644923cd427d1 P df6733a6ac4a4b7b87437920eaa15585: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 09:58:59.708514 15293 tablet_replica.cc:330] T e4b9ffaa1adf4096995644923cd427d1 P d540747f7bab4b0aa074cda755e1642d: stopping tablet replica
I20260924 09:58:59.708899 15293 raft_consensus.cc:2229] T e4b9ffaa1adf4096995644923cd427d1 P d540747f7bab4b0aa074cda755e1642d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 09:58:59.709486 15293 raft_consensus.cc:2258] T e4b9ffaa1adf4096995644923cd427d1 P d540747f7bab4b0aa074cda755e1642d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 09:58:59.710094 15294 ts_tablet_manager.cc:1905] T e4b9ffaa1adf4096995644923cd427d1 P fc45f069420445869c886224da489dc6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 09:58:59.711380 15293 ts_tablet_manager.cc:1905] T e4b9ffaa1adf4096995644923cd427d1 P d540747f7bab4b0aa074cda755e1642d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 09:58:59.711912 15263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:58:59.691142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44524 (local address 127.0.97.1:36377)
0924 09:58:59.691345 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0924 09:58:59.691351 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 09:58:59.691367 (+ 16us) server_negotiation.cc:407] Connection header received
0924 09:58:59.691423 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:58:59.691429 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:58:59.691521 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:58:59.691640 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:58:59.693796 (+ 2156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:58:59.694962 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:58:59.696920 (+ 1958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:58:59.697611 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:58:59.697728 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:58:59.703836 (+ 6108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:58:59.703852 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:58:59.703870 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:58:59.703912 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:58:59.707902 (+ 3990us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:58:59.708810 (+ 908us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:58:59.708819 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:58:59.708822 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:58:59.708966 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:58:59.709424 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:58:59.709429 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:58:59.709432 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:58:59.709878 (+ 446us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:58:59.710140 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0924 09:58:59.711309 (+ 1169us) server_negotiation.cc:299] Negotiation successful
0924 09:58:59.711554 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 09:58:59.712576 14985 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 09:58:59.716506 15295 ts_tablet_manager.cc:1918] T e4b9ffaa1adf4096995644923cd427d1 P df6733a6ac4a4b7b87437920eaa15585: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 09:58:59.716743 15295 log.cc:1196] T e4b9ffaa1adf4096995644923cd427d1 P df6733a6ac4a4b7b87437920eaa15585: Deleting WAL directory at /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/minicluster-data/ts-2/wal/wals/e4b9ffaa1adf4096995644923cd427d1
I20260924 09:58:59.717334 15295 ts_tablet_manager.cc:1939] T e4b9ffaa1adf4096995644923cd427d1 P df6733a6ac4a4b7b87437920eaa15585: Deleting consensus metadata
I20260924 09:58:59.717864 15293 ts_tablet_manager.cc:1918] T e4b9ffaa1adf4096995644923cd427d1 P d540747f7bab4b0aa074cda755e1642d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 09:58:59.718001 15293 log.cc:1196] T e4b9ffaa1adf4096995644923cd427d1 P d540747f7bab4b0aa074cda755e1642d: Deleting WAL directory at /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/minicluster-data/ts-1/wal/wals/e4b9ffaa1adf4096995644923cd427d1
I20260924 09:58:59.718755 15293 ts_tablet_manager.cc:1939] T e4b9ffaa1adf4096995644923cd427d1 P d540747f7bab4b0aa074cda755e1642d: Deleting consensus metadata
I20260924 09:58:59.719512 14791 catalog_manager.cc:4841] TS df6733a6ac4a4b7b87437920eaa15585 (127.0.97.3:39885): tablet e4b9ffaa1adf4096995644923cd427d1 (table test-table [id=73006212c0a148f596f01ad14d5d3bf8]) successfully deleted
I20260924 09:58:59.720372 14790 catalog_manager.cc:4841] TS d540747f7bab4b0aa074cda755e1642d (127.0.97.2:40223): tablet e4b9ffaa1adf4096995644923cd427d1 (table test-table [id=73006212c0a148f596f01ad14d5d3bf8]) successfully deleted
I20260924 09:58:59.802814 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu with pid 14842
W20260924 09:58:59.825040 14792 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:36377 (error 108)
I20260924 09:58:59.825336 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu pid 14994
I20260924 09:58:59.827111 15264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:58:59.826650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36377 (local address 127.0.0.1:44528)
0924 09:58:59.826829 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0924 09:58:59.826940 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:36377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260924 09:58:59.827456 14792 catalog_manager.cc:4584] TS fc45f069420445869c886224da489dc6 (127.0.97.1:36377): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e4b9ffaa1adf4096995644923cd427d1: Network error: Client connection negotiation failed: client connection to 127.0.97.1:36377: connect: Connection refused (error 111)
I20260924 09:58:59.842972 15266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:58:59.826750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36882 (local address 127.0.97.2:40223)
0924 09:58:59.826916 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0924 09:58:59.826922 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 09:58:59.826942 (+ 20us) server_negotiation.cc:407] Connection header received
0924 09:58:59.827004 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:58:59.827009 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:58:59.827093 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:58:59.827201 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:58:59.828765 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:58:59.829511 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:58:59.830777 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:58:59.831168 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:58:59.831304 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:58:59.837623 (+ 6319us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:58:59.837638 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:58:59.837653 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:58:59.837695 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:58:59.840375 (+ 2680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:58:59.841017 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:58:59.841025 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:58:59.841027 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:58:59.841145 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:58:59.841507 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:58:59.841510 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:58:59.841512 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:58:59.841874 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:58:59.842145 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0924 09:58:59.842439 (+ 294us) server_negotiation.cc:299] Negotiation successful
0924 09:58:59.842641 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260924 09:58:59.843752 15117 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 09:58:59.881132 15264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:58:59.880605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36377 (local address 127.0.0.1:44532)
0924 09:58:59.880832 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0924 09:58:59.880946 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:36377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260924 09:58:59.882208 15264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:58:59.881825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36377 (local address 127.0.0.1:44536)
0924 09:58:59.881973 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0924 09:58:59.882058 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:36377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260924 09:58:59.931023 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu with pid 14994
I20260924 09:58:59.942835 15264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:58:59.942272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36377 (local address 127.0.0.1:44540)
0924 09:58:59.942523 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0924 09:58:59.942625 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:36377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260924 09:58:59.944020 15264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:58:59.943630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36377 (local address 127.0.0.1:44554)
0924 09:58:59.943796 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0924 09:58:59.943886 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:36377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260924 09:58:59.945614 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu pid 15125
I20260924 09:58:59.960654 15265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:58:59.946356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60178 (local address 127.0.97.3:39885)
0924 09:58:59.946784 (+ 428us) server_negotiation.cc:206] Beginning negotiation
0924 09:58:59.946789 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 09:58:59.946808 (+ 19us) server_negotiation.cc:407] Connection header received
0924 09:58:59.946868 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:58:59.946873 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:58:59.946957 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:58:59.947056 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:58:59.948391 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:58:59.949599 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:58:59.950896 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:58:59.951692 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:58:59.951828 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:58:59.954088 (+ 2260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:58:59.954103 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:58:59.954123 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:58:59.954174 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:58:59.957863 (+ 3689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:58:59.958512 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:58:59.958521 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:58:59.958525 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:58:59.958705 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:58:59.959083 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:58:59.959087 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:58:59.959089 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:58:59.959531 (+ 442us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:58:59.959875 (+ 344us) server_negotiation.cc:1035] Waiting for connection context
0924 09:58:59.960108 (+ 233us) server_negotiation.cc:299] Negotiation successful
0924 09:58:59.960321 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":305}
I20260924 09:58:59.961367 15248 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 09:59:00.024789 15264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:59:00.024362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36377 (local address 127.0.0.1:44558)
0924 09:59:00.024541 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0924 09:59:00.024640 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:36377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260924 09:59:00.025730 15264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:59:00.025394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36377 (local address 127.0.0.1:44560)
0924 09:59:00.025566 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0924 09:59:00.025629 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:36377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260924 09:59:00.079519 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu with pid 15125
I20260924 09:59:00.095651 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu pid 14772
I20260924 09:59:00.096712 14835 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 09:59:00.207547 15264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:59:00.206976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36377 (local address 127.0.0.1:44562)
0924 09:59:00.207135 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0924 09:59:00.207285 (+ 150us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:36377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260924 09:59:00.207855 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu with pid 14772
I20260924 09:59:00.224871 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
/tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46551
--webserver_interface=127.0.97.62
--webserver_port=38575
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:46551
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 09:59:00.539578 15301 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 09:59:00.539868 15301 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 09:59:00.539965 15301 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 09:59:00.547065 15301 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 09:59:00.547166 15301 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 09:59:00.547217 15301 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 09:59:00.547290 15301 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 09:59:00.547333 15301 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 09:59:00.557029 15301 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:46551
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46551
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=38575
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 09:59:00.558292 15301 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 09:59:00.559547 15301 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 09:59:00.570683 15306 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:59:00.570619 15307 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:59:00.571074 15309 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 09:59:00.571308 15301 server_base.cc:1034] running on GCE node
I20260924 09:59:00.572075 15301 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 09:59:00.573529 15301 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 09:59:00.574728 15301 hybrid_clock.cc:648] HybridClock initialized: now 1790243940574669 us; error 49 us; skew 500 ppm
I20260924 09:59:00.575067 15301 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 09:59:00 dist-test-slave-w8d2 krb5kdc[14221](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243940, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 09:59:00.581772 15301 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260924 09:59:00.583338 15301 webserver.cc:460] Webserver started at http://127.0.97.62:38575/ using document root <none> and password file <none>
I20260924 09:59:00.584086 15301 fs_manager.cc:359] Metadata directory not provided
I20260924 09:59:00.584205 15301 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 09:59:00.589788 15301 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 09:59:00.592717 15316 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 09:59:00.594113 15301 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 09:59:00.594281 15301 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/minicluster-data/master-0/wal
uuid: "76cdda25f78f4f8dab4b5e8cdf63a2b7"
format_stamp: "Formatted at 2026-09-24 09:58:55 on dist-test-slave-w8d2"
server_key: "455e524ee257bc375ab1961387ac0eca"
server_key_iv: "8c564b7cccd5eb633989057664a5c375"
server_key_version: "encryptionkey@0"
I20260924 09:59:00.595278 15301 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 09:59:00.607030 15301 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 09:59:00.610947 15301 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 09:59:00.611204 15301 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 09:59:00.628290 15301 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:46551
I20260924 09:59:00.628314 15367 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:46551 every 8 connection(s)
I20260924 09:59:00.629798 15301 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/minicluster-data/master-0/data/info.pb
I20260924 09:59:00.635937 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu as pid 15301
I20260924 09:59:00.636694 15368 sys_catalog.cc:263] Verifying existing consensus state
I20260924 09:59:00.636687 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
/tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:36377
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:46551
--webserver_port=44965
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 09:59:00.639946 15368 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76cdda25f78f4f8dab4b5e8cdf63a2b7: Bootstrap starting.
I20260924 09:59:00.659600 15368 log.cc:824] T 00000000000000000000000000000000 P 76cdda25f78f4f8dab4b5e8cdf63a2b7: Log is configured to *not* fsync() on all Append() calls
I20260924 09:59:00.675379 15368 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76cdda25f78f4f8dab4b5e8cdf63a2b7: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 09:59:00.675886 15368 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76cdda25f78f4f8dab4b5e8cdf63a2b7: Bootstrap complete.
I20260924 09:59:00.684651 15368 raft_consensus.cc:348] T 00000000000000000000000000000000 P 76cdda25f78f4f8dab4b5e8cdf63a2b7 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76cdda25f78f4f8dab4b5e8cdf63a2b7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46551 } }
I20260924 09:59:00.685509 15368 raft_consensus.cc:729] T 00000000000000000000000000000000 P 76cdda25f78f4f8dab4b5e8cdf63a2b7 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76cdda25f78f4f8dab4b5e8cdf63a2b7, State: Initialized, Role: FOLLOWER
I20260924 09:59:00.686086 15368 consensus_queue.cc:260] T 00000000000000000000000000000000 P 76cdda25f78f4f8dab4b5e8cdf63a2b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76cdda25f78f4f8dab4b5e8cdf63a2b7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46551 } }
I20260924 09:59:00.686288 15368 raft_consensus.cc:388] T 00000000000000000000000000000000 P 76cdda25f78f4f8dab4b5e8cdf63a2b7 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 09:59:00.686399 15368 raft_consensus.cc:482] T 00000000000000000000000000000000 P 76cdda25f78f4f8dab4b5e8cdf63a2b7 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 09:59:00.686554 15368 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 76cdda25f78f4f8dab4b5e8cdf63a2b7 [term 2 FOLLOWER]: Advancing to term 3
I20260924 09:59:00.690238 15368 raft_consensus.cc:504] T 00000000000000000000000000000000 P 76cdda25f78f4f8dab4b5e8cdf63a2b7 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76cdda25f78f4f8dab4b5e8cdf63a2b7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46551 } }
I20260924 09:59:00.690709 15368 leader_election.cc:302] T 00000000000000000000000000000000 P 76cdda25f78f4f8dab4b5e8cdf63a2b7 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 76cdda25f78f4f8dab4b5e8cdf63a2b7; no voters:
I20260924 09:59:00.691335 15368 leader_election.cc:288] T 00000000000000000000000000000000 P 76cdda25f78f4f8dab4b5e8cdf63a2b7 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260924 09:59:00.691469 15372 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 76cdda25f78f4f8dab4b5e8cdf63a2b7 [term 3 FOLLOWER]: Leader election won for term 3
I20260924 09:59:00.693535 15368 sys_catalog.cc:564] T 00000000000000000000000000000000 P 76cdda25f78f4f8dab4b5e8cdf63a2b7 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 09:59:00.695498 15372 raft_consensus.cc:686] T 00000000000000000000000000000000 P 76cdda25f78f4f8dab4b5e8cdf63a2b7 [term 3 LEADER]: Becoming Leader. State: Replica: 76cdda25f78f4f8dab4b5e8cdf63a2b7, State: Running, Role: LEADER
I20260924 09:59:00.696161 15372 consensus_queue.cc:237] T 00000000000000000000000000000000 P 76cdda25f78f4f8dab4b5e8cdf63a2b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76cdda25f78f4f8dab4b5e8cdf63a2b7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46551 } }
I20260924 09:59:00.698849 15374 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76cdda25f78f4f8dab4b5e8cdf63a2b7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 76cdda25f78f4f8dab4b5e8cdf63a2b7. Latest consensus state: current_term: 3 leader_uuid: "76cdda25f78f4f8dab4b5e8cdf63a2b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76cdda25f78f4f8dab4b5e8cdf63a2b7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46551 } } }
I20260924 09:59:00.699275 15374 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76cdda25f78f4f8dab4b5e8cdf63a2b7 [sys.catalog]: This master's current role is: LEADER
I20260924 09:59:00.699796 15373 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76cdda25f78f4f8dab4b5e8cdf63a2b7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "76cdda25f78f4f8dab4b5e8cdf63a2b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76cdda25f78f4f8dab4b5e8cdf63a2b7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46551 } } }
I20260924 09:59:00.700055 15373 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76cdda25f78f4f8dab4b5e8cdf63a2b7 [sys.catalog]: This master's current role is: LEADER
I20260924 09:59:00.713310 15382 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 09:59:00.721560 15382 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 09:59:00.723089 15382 catalog_manager.cc:1259] Loaded cluster ID: e58d63fbf0b544da81d1e5cd9e6c8806
I20260924 09:59:00.723281 15382 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260924 09:59:00.725311 15382 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f31d9d6b9ad google::LogMessage::Fail() at ??:0
@ 0x7f31d9d6e88a google::LogMessage::SendToLog() at ??:0
@ 0x7f31d9d6b4de google::LogMessage::Flush() at ??:0
@ 0x7f31d9d6ca19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f31ef7a2512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f31ef79f976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f31ef827610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f31ef827439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f31dc9b7345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f31dc9ba91c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f31dc9ba799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f31dc995eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f31dfc6f6db start_thread at ??:0
@ 0x7f31d8c0561f clone at ??:0
W20260924 09:59:00.984212 15370 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 09:59:00.984514 15370 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 09:59:00.984650 15370 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 09:59:00.992040 15370 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 09:59:00.992159 15370 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 09:59:00.992302 15370 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260924 09:59:01.002200 15370 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:36377
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=44965
--tserver_master_addrs=127.0.97.62:46551
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 09:59:01.003571 15370 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 09:59:01.004937 15370 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 09:59:01.017590 15409 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:59:01.017660 15410 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:59:01.017628 15412 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 09:59:01.018450 15370 server_base.cc:1034] running on GCE node
I20260924 09:59:01.019052 15370 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 09:59:01.019984 15370 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 09:59:01.021186 15370 hybrid_clock.cc:648] HybridClock initialized: now 1790243941021130 us; error 43 us; skew 500 ppm
I20260924 09:59:01.021482 15370 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 09:59:01 dist-test-slave-w8d2 krb5kdc[14221](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243941, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 09:59:01.024379 15370 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260924 09:59:01.025833 15370 webserver.cc:460] Webserver started at http://127.0.97.1:44965/ using document root <none> and password file <none>
I20260924 09:59:01.026490 15370 fs_manager.cc:359] Metadata directory not provided
I20260924 09:59:01.026607 15370 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 09:59:01.031826 15370 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260924 09:59:01.034559 15419 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 09:59:01.035876 15370 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 09:59:01.036037 15370 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/minicluster-data/ts-0/wal
uuid: "fc45f069420445869c886224da489dc6"
format_stamp: "Formatted at 2026-09-24 09:58:55 on dist-test-slave-w8d2"
server_key: "49da492ccbc870cadd378684801b7529"
server_key_iv: "82c3f8a4747d352eaf96db4cf43ace10"
server_key_version: "encryptionkey@0"
I20260924 09:59:01.037029 15370 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 09:59:01.055483 15370 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 09:59:01.059612 15370 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 09:59:01.059864 15370 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 09:59:01.060830 15370 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 09:59:01.062580 15426 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
W20260924 09:59:01.067900 15426 ts_tablet_manager.cc:1852] T e4b9ffaa1adf4096995644923cd427d1 P fc45f069420445869c886224da489dc6: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260924 09:59:01.068868 15426 ts_tablet_manager.cc:1905] T e4b9ffaa1adf4096995644923cd427d1 P fc45f069420445869c886224da489dc6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 09:59:01.075999 15426 ts_tablet_manager.cc:1918] T e4b9ffaa1adf4096995644923cd427d1 P fc45f069420445869c886224da489dc6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 09:59:01.076175 15426 log.cc:1196] T e4b9ffaa1adf4096995644923cd427d1 P fc45f069420445869c886224da489dc6: Deleting WAL directory at /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/minicluster-data/ts-0/wal/wals/e4b9ffaa1adf4096995644923cd427d1
I20260924 09:59:01.076619 15426 ts_tablet_manager.cc:1939] T e4b9ffaa1adf4096995644923cd427d1 P fc45f069420445869c886224da489dc6: Deleting consensus metadata
I20260924 09:59:01.076983 15370 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260924 09:59:01.077098 15370 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.015s user 0.001s sys 0.000s
I20260924 09:59:01.077208 15370 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 09:59:01.077275 15370 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 09:59:01.106990 15370 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:36377
I20260924 09:59:01.107039 15532 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:36377 every 8 connection(s)
I20260924 09:59:01.108278 15370 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/minicluster-data/ts-0/data/info.pb
I20260924 09:59:01.112782 15535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:59:01.111025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46551 (local address 127.0.97.1:35925)
0924 09:59:01.111740 (+ 715us) negotiation.cc:107] Waiting for socket to connect
0924 09:59:01.111943 (+ 203us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":284,"threads_started":1}
W20260924 09:59:01.113358 15533 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:46551 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:46551: Client connection negotiation failed: client connection to 127.0.97.62:46551: connect: Connection refused (error 111)
I20260924 09:59:01.114050 15535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:59:01.113775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46551 (local address 127.0.97.1:33065)
0924 09:59:01.113896 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0924 09:59:01.113957 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260924 09:59:01.114758 15535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:59:01.114491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46551 (local address 127.0.97.1:52033)
0924 09:59:01.114588 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0924 09:59:01.114653 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260924 09:59:01.114935 15533 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 09:59:01.118320 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu as pid 15370
I20260924 09:59:01.118873 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
/tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:40223
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:46551
--webserver_port=44365
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 09:59:01.431309 15536 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 09:59:01.431618 15536 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 09:59:01.431742 15536 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 09:59:01.439064 15536 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 09:59:01.439193 15536 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 09:59:01.439364 15536 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260924 09:59:01.449232 15536 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:40223
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=44365
--tserver_master_addrs=127.0.97.62:46551
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 09:59:01.450532 15536 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 09:59:01.451894 15536 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 09:59:01.464846 15542 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:59:01.464954 15544 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:59:01.464936 15541 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 09:59:01.468734 15536 server_base.cc:1034] running on GCE node
I20260924 09:59:01.469405 15536 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 09:59:01.470372 15536 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 09:59:01.471568 15536 hybrid_clock.cc:648] HybridClock initialized: now 1790243941471521 us; error 44 us; skew 500 ppm
I20260924 09:59:01.471874 15536 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 09:59:01 dist-test-slave-w8d2 krb5kdc[14221](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243941, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 09:59:01.474871 15536 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260924 09:59:01.476444 15536 webserver.cc:460] Webserver started at http://127.0.97.2:44365/ using document root <none> and password file <none>
I20260924 09:59:01.477150 15536 fs_manager.cc:359] Metadata directory not provided
I20260924 09:59:01.477268 15536 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 09:59:01.482492 15536 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.002s
I20260924 09:59:01.485201 15551 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 09:59:01.486485 15536 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 09:59:01.486634 15536 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/minicluster-data/ts-1/wal
uuid: "d540747f7bab4b0aa074cda755e1642d"
format_stamp: "Formatted at 2026-09-24 09:58:56 on dist-test-slave-w8d2"
server_key: "0fa54e7f8d9fcddc5e832038d95a6d98"
server_key_iv: "3bb8091213b5eacbce3be064ffdb49ec"
server_key_version: "encryptionkey@0"
I20260924 09:59:01.487620 15536 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 09:59:01.500908 15536 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 09:59:01.504796 15536 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 09:59:01.505048 15536 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 09:59:01.505966 15536 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 09:59:01.507304 15536 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 09:59:01.507395 15536 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 09:59:01.507500 15536 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 09:59:01.507570 15536 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 09:59:01.537132 15536 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:40223
I20260924 09:59:01.537223 15663 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:40223 every 8 connection(s)
I20260924 09:59:01.538534 15536 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/minicluster-data/ts-1/data/info.pb
I20260924 09:59:01.539057 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu as pid 15536
I20260924 09:59:01.539819 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
/tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:39885
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:46551
--webserver_port=41853
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 09:59:01.543859 15666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:59:01.541971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46551 (local address 127.0.97.2:51563)
0924 09:59:01.542737 (+ 766us) negotiation.cc:107] Waiting for socket to connect
0924 09:59:01.542909 (+ 172us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":295,"threads_started":1}
W20260924 09:59:01.544552 15664 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:46551 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:46551: Client connection negotiation failed: client connection to 127.0.97.62:46551: connect: Connection refused (error 111)
I20260924 09:59:01.545323 15666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:59:01.545018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46551 (local address 127.0.97.2:60645)
0924 09:59:01.545158 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0924 09:59:01.545227 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260924 09:59:01.546159 15666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:59:01.545827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46551 (local address 127.0.97.2:36849)
0924 09:59:01.545963 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0924 09:59:01.546056 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260924 09:59:01.546389 15664 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 09:59:01.849201 15667 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 09:59:01.849469 15667 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 09:59:01.849591 15667 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 09:59:01.856652 15667 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 09:59:01.856748 15667 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 09:59:01.856899 15667 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260924 09:59:01.866312 15667 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:39885
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=41853
--tserver_master_addrs=127.0.97.62:46551
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 09:59:01.867553 15667 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 09:59:01.868793 15667 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 09:59:01.880798 15672 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:59:01.880826 15675 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:59:01.880761 15673 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 09:59:01.881546 15667 server_base.cc:1034] running on GCE node
I20260924 09:59:01.882119 15667 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 09:59:01.883003 15667 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 09:59:01.884196 15667 hybrid_clock.cc:648] HybridClock initialized: now 1790243941884152 us; error 30 us; skew 500 ppm
I20260924 09:59:01.884490 15667 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 09:59:01 dist-test-slave-w8d2 krb5kdc[14221](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243941, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 09:59:01.887424 15667 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260924 09:59:01.888705 15667 webserver.cc:460] Webserver started at http://127.0.97.3:41853/ using document root <none> and password file <none>
I20260924 09:59:01.889369 15667 fs_manager.cc:359] Metadata directory not provided
I20260924 09:59:01.889482 15667 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 09:59:01.894831 15667 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260924 09:59:01.897536 15682 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 09:59:01.898798 15667 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260924 09:59:01.898963 15667 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/minicluster-data/ts-2/wal
uuid: "df6733a6ac4a4b7b87437920eaa15585"
format_stamp: "Formatted at 2026-09-24 09:58:56 on dist-test-slave-w8d2"
server_key: "eb52ee37941673b737f37b0fa0e5abe5"
server_key_iv: "2116980e992b0b4705d1395bb478eccc"
server_key_version: "encryptionkey@0"
I20260924 09:59:01.899960 15667 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 09:59:01.925901 15667 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 09:59:01.930017 15667 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 09:59:01.930256 15667 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 09:59:01.931162 15667 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 09:59:01.932391 15667 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 09:59:01.932480 15667 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 09:59:01.932578 15667 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 09:59:01.932644 15667 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 09:59:01.961191 15667 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:39885
I20260924 09:59:01.961254 15794 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:39885 every 8 connection(s)
I20260924 09:59:01.962579 15667 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790243735634835-388-0/minicluster-data/ts-2/data/info.pb
I20260924 09:59:01.967084 15797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:59:01.965363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46551 (local address 127.0.97.3:50239)
0924 09:59:01.966047 (+ 684us) negotiation.cc:107] Waiting for socket to connect
0924 09:59:01.966243 (+ 196us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":259,"threads_started":1}
W20260924 09:59:01.967697 15795 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:46551 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:46551: Client connection negotiation failed: client connection to 127.0.97.62:46551: connect: Connection refused (error 111)
I20260924 09:59:01.968468 15797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:59:01.968190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46551 (local address 127.0.97.3:35951)
0924 09:59:01.968310 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0924 09:59:01.968372 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260924 09:59:01.969187 15797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:59:01.968917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46551 (local address 127.0.97.3:46881)
0924 09:59:01.969014 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0924 09:59:01.969091 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260924 09:59:01.969376 15795 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 09:59:01.970036 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu as pid 15667
I20260924 09:59:02.116195 15799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:59:02.115470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46551 (local address 127.0.97.1:48191)
0924 09:59:02.115893 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0924 09:59:02.115974 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":202,"threads_started":1}
I20260924 09:59:02.547637 15800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:59:02.546848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46551 (local address 127.0.97.2:54885)
0924 09:59:02.547332 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0924 09:59:02.547422 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":236,"threads_started":1}
I20260924 09:59:02.970736 15801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:59:02.969914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46551 (local address 127.0.97.3:42513)
0924 09:59:02.970391 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0924 09:59:02.970476 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":250,"threads_started":1}
I20260924 09:59:03.117609 15802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:59:03.116973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46551 (local address 127.0.97.1:41225)
0924 09:59:03.117380 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0924 09:59:03.117458 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":215,"threads_started":1}
I20260924 09:59:03.549129 15803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:59:03.548444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46551 (local address 127.0.97.2:34563)
0924 09:59:03.548854 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0924 09:59:03.548941 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":201,"threads_started":1}
I20260924 09:59:03.972353 15804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:59:03.971603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46551 (local address 127.0.97.3:49641)
0924 09:59:03.972057 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0924 09:59:03.972176 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":221,"threads_started":1}
I20260924 09:59:04.119100 15805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:59:04.118426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46551 (local address 127.0.97.1:44539)
0924 09:59:04.118853 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0924 09:59:04.118937 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":225,"threads_started":1}
I20260924 09:59:04.550668 15806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:59:04.549974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46551 (local address 127.0.97.2:53909)
0924 09:59:04.550419 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0924 09:59:04.550502 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":234,"threads_started":1}
I20260924 09:59:04.974042 15807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:59:04.973309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46551 (local address 127.0.97.3:59113)
0924 09:59:04.973788 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0924 09:59:04.973870 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":255,"threads_started":1}
I20260924 09:59:05.120728 15808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:59:05.120019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46551 (local address 127.0.97.1:33713)
0924 09:59:05.120484 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0924 09:59:05.120566 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":229,"threads_started":1}
I20260924 09:59:05.552490 15809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:59:05.551716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46551 (local address 127.0.97.2:37705)
0924 09:59:05.552191 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0924 09:59:05.552280 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":232,"threads_started":1}
I20260924 09:59:05.975600 15810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:59:05.974879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46551 (local address 127.0.97.3:34535)
0924 09:59:05.975325 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0924 09:59:05.975409 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":198,"threads_started":1}
I20260924 09:59:06.122282 15811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:59:06.121608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46551 (local address 127.0.97.1:34987)
0924 09:59:06.122021 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0924 09:59:06.122111 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":200,"threads_started":1}
I20260924 09:59:06.554075 15812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:59:06.553375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46551 (local address 127.0.97.2:60529)
0924 09:59:06.553804 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0924 09:59:06.553905 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":206,"threads_started":1}
I20260924 09:59:06.977408 15813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:59:06.976507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46551 (local address 127.0.97.3:57565)
0924 09:59:06.977103 (+ 596us) negotiation.cc:107] Waiting for socket to connect
0924 09:59:06.977205 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":286,"threads_started":1}
I20260924 09:59:07.123790 15814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:59:07.123096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46551 (local address 127.0.97.1:54811)
0924 09:59:07.123529 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0924 09:59:07.123624 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":247,"threads_started":1}
I20260924 09:59:07.555910 15815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:59:07.555069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46551 (local address 127.0.97.2:59833)
0924 09:59:07.555609 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0924 09:59:07.555722 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":297,"threads_started":1}
I20260924 09:59:07.979113 15816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:59:07.978387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46551 (local address 127.0.97.3:51431)
0924 09:59:07.978823 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0924 09:59:07.978938 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":219,"threads_started":1}
I20260924 09:59:08.125501 15817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:59:08.124703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46551 (local address 127.0.97.1:39083)
0924 09:59:08.125208 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0924 09:59:08.125292 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":224,"threads_started":1}
I20260924 09:59:08.557626 15818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:59:08.556806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46551 (local address 127.0.97.2:44609)
0924 09:59:08.557323 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0924 09:59:08.557409 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":234,"threads_started":1}
I20260924 09:59:08.980868 15819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:59:08.980093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46551 (local address 127.0.97.3:50253)
0924 09:59:08.980574 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0924 09:59:08.980658 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":203,"threads_started":1}
I20260924 09:59:09.127265 15820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:59:09.126498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46551 (local address 127.0.97.1:52079)
0924 09:59:09.126978 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0924 09:59:09.127064 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":241,"threads_started":1}
I20260924 09:59:09.559305 15821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:59:09.558553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46551 (local address 127.0.97.2:48117)
0924 09:59:09.559016 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0924 09:59:09.559102 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":236,"threads_started":1}
I20260924 09:59:09.982646 15822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:59:09.981816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46551 (local address 127.0.97.3:40629)
0924 09:59:09.982355 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0924 09:59:09.982446 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":275,"threads_started":1}
I20260924 09:59:10.128839 15823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:59:10.128089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46551 (local address 127.0.97.1:43331)
0924 09:59:10.128557 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0924 09:59:10.128655 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":241,"threads_started":1}
I20260924 09:59:10.560855 15824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:59:10.560153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46551 (local address 127.0.97.2:49771)
0924 09:59:10.560603 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0924 09:59:10.560685 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":203,"threads_started":1}
I20260924 09:59:10.984272 15825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:59:10.983516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46551 (local address 127.0.97.3:51447)
0924 09:59:10.984015 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0924 09:59:10.984100 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":228,"threads_started":1}
I20260924 09:59:11.130450 15826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:59:11.129766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46551 (local address 127.0.97.1:34755)
0924 09:59:11.130200 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0924 09:59:11.130284 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":227,"threads_started":1}
I20260924 09:59:11.562445 15827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:59:11.561736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46551 (local address 127.0.97.2:39589)
0924 09:59:11.562206 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0924 09:59:11.562289 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":264,"threads_started":1}
I20260924 09:59:11.985787 15828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:59:11.985117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46551 (local address 127.0.97.3:33101)
0924 09:59:11.985543 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0924 09:59:11.985625 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":228,"threads_started":1}
I20260924 09:59:12.131981 15829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:59:12.131319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46551 (local address 127.0.97.1:35241)
0924 09:59:12.131731 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0924 09:59:12.131807 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":217,"threads_started":1}
I20260924 09:59:12.563995 15830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:59:12.563321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46551 (local address 127.0.97.2:50623)
0924 09:59:12.563734 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0924 09:59:12.563825 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":205,"threads_started":1}
I20260924 09:59:12.987493 15831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:59:12.986752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46551 (local address 127.0.97.3:57183)
0924 09:59:12.987166 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0924 09:59:12.987318 (+ 152us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":196,"threads_started":1}
I20260924 09:59:13.133419 15832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:59:13.132726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46551 (local address 127.0.97.1:53535)
0924 09:59:13.133179 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0924 09:59:13.133258 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":242,"threads_started":1}
I20260924 09:59:13.565574 15833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:59:13.564903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46551 (local address 127.0.97.2:56259)
0924 09:59:13.565338 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0924 09:59:13.565420 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":211,"threads_started":1}
I20260924 09:59:13.989032 15834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:59:13.988348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46551 (local address 127.0.97.3:46955)
0924 09:59:13.988784 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0924 09:59:13.988879 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":228,"threads_started":1}
I20260924 09:59:14.134951 15835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:59:14.134248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46551 (local address 127.0.97.1:49853)
0924 09:59:14.134666 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0924 09:59:14.134744 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":219,"threads_started":1}
I20260924 09:59:14.567119 15836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:59:14.566422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46551 (local address 127.0.97.2:57003)
0924 09:59:14.566849 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0924 09:59:14.566934 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":210,"threads_started":1}
I20260924 09:59:14.990525 15837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:59:14.989820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46551 (local address 127.0.97.3:56525)
0924 09:59:14.990259 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0924 09:59:14.990341 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":207,"threads_started":1}
I20260924 09:59:15.136394 15838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:59:15.135681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46551 (local address 127.0.97.1:46959)
0924 09:59:15.136138 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0924 09:59:15.136216 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":238,"threads_started":1}
I20260924 09:59:15.568675 15839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:59:15.567992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46551 (local address 127.0.97.2:59723)
0924 09:59:15.568393 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0924 09:59:15.568499 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":194,"threads_started":1}
I20260924 09:59:15.992075 15840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:59:15.991355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46551 (local address 127.0.97.3:45503)
0924 09:59:15.991802 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0924 09:59:15.991878 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":243,"threads_started":1}
I20260924 09:59:16.138105 15841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:59:16.137407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46551 (local address 127.0.97.1:58917)
0924 09:59:16.137845 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0924 09:59:16.137925 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":213,"threads_started":1}
I20260924 09:59:16.570403 15842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:59:16.569664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46551 (local address 127.0.97.2:60359)
0924 09:59:16.570124 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0924 09:59:16.570219 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":231,"threads_started":1}
I20260924 09:59:16.975742 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu pid 15370
I20260924 09:59:16.992508 15843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:59:16.976708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60298 (local address 127.0.97.1:36377)
0924 09:59:16.977238 (+ 530us) server_negotiation.cc:206] Beginning negotiation
0924 09:59:16.977253 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 09:59:16.977288 (+ 35us) server_negotiation.cc:407] Connection header received
0924 09:59:16.979019 (+ 1731us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:59:16.979029 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:59:16.979521 (+ 492us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:59:16.979808 (+ 287us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:59:16.981024 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:59:16.982274 (+ 1250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:59:16.983458 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:59:16.984120 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:59:16.984256 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:59:16.986659 (+ 2403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:59:16.986666 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:59:16.986681 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:59:16.986724 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:59:16.989538 (+ 2814us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:59:16.990115 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:59:16.990122 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:59:16.990128 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:59:16.990292 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:59:16.990580 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:59:16.990584 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:59:16.990586 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:59:16.991186 (+ 600us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:59:16.991463 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0924 09:59:16.991788 (+ 325us) server_negotiation.cc:299] Negotiation successful
0924 09:59:16.992223 (+ 435us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":227,"threads_started":1}
I20260924 09:59:16.993762 15844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:59:16.993097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46551 (local address 127.0.97.3:57789)
0924 09:59:16.993527 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0924 09:59:16.993606 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":222,"threads_started":1}
I20260924 09:59:16.994047 15528 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 09:59:17.075274 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu with pid 15370
I20260924 09:59:17.089358 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu pid 15536
I20260924 09:59:17.106679 15846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:59:17.090349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58996 (local address 127.0.97.2:40223)
0924 09:59:17.090878 (+ 529us) server_negotiation.cc:206] Beginning negotiation
0924 09:59:17.090893 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 09:59:17.090949 (+ 56us) server_negotiation.cc:407] Connection header received
0924 09:59:17.092654 (+ 1705us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:59:17.092666 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:59:17.093098 (+ 432us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:59:17.093432 (+ 334us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:59:17.094916 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:59:17.096177 (+ 1261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:59:17.097433 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:59:17.098093 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:59:17.098289 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:59:17.100786 (+ 2497us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:59:17.100794 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:59:17.100809 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:59:17.100851 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:59:17.103558 (+ 2707us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:59:17.104215 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:59:17.104224 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:59:17.104231 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:59:17.104403 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:59:17.104739 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:59:17.104744 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:59:17.104747 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:59:17.105250 (+ 503us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:59:17.105524 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0924 09:59:17.105978 (+ 454us) server_negotiation.cc:299] Negotiation successful
0924 09:59:17.106384 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":382,"thread_start_us":251,"threads_started":1}
I20260924 09:59:17.108129 15659 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 09:59:17.189397 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu with pid 15536
I20260924 09:59:17.203703 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu pid 15667
I20260924 09:59:17.220072 15848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:59:17.204590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46926 (local address 127.0.97.3:39885)
0924 09:59:17.205166 (+ 576us) server_negotiation.cc:206] Beginning negotiation
0924 09:59:17.205180 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 09:59:17.205219 (+ 39us) server_negotiation.cc:407] Connection header received
0924 09:59:17.206778 (+ 1559us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:59:17.206788 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:59:17.207198 (+ 410us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:59:17.207668 (+ 470us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:59:17.209042 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:59:17.210241 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:59:17.211441 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:59:17.212066 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:59:17.212240 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:59:17.214610 (+ 2370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:59:17.214618 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:59:17.214632 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:59:17.214672 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:59:17.217133 (+ 2461us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:59:17.217733 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:59:17.217739 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:59:17.217745 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:59:17.217915 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:59:17.218251 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:59:17.218255 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:59:17.218257 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:59:17.218731 (+ 474us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:59:17.219001 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0924 09:59:17.219421 (+ 420us) server_negotiation.cc:299] Negotiation successful
0924 09:59:17.219785 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":424,"thread_start_us":287,"threads_started":1}
I20260924 09:59:17.221354 15790 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 09:59:17.301471 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu with pid 15667
2026-09-24T09:59:17Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (23644 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 09:59:17 dist-test-slave-w8d2 krb5kdc[15852](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 09:59:17 dist-test-slave-w8d2 krb5kdc[15852](info): set up 2 sockets
Sep 24 09:59:17 dist-test-slave-w8d2 krb5kdc[15852](info): commencing operation
krb5kdc: starting...
W20260924 09:59:18.311542 388 mini_kdc.cc:121] Time spent starting KDC: real 0.975s user 0.006s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/krb5kdc/test-user.keytab.
Sep 24 09:59:18 dist-test-slave-w8d2 krb5kdc[15852](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243958, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T09:59:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T09:59:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260924 09:59:18.522440 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
/tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35771
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:35771
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 09:59:18.831513 15865 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 09:59:18.831816 15865 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 09:59:18.831948 15865 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 09:59:18.839190 15865 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260924 09:59:18.839315 15865 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260924 09:59:18.839372 15865 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260924 09:59:18.839422 15865 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 09:59:18.839473 15865 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 09:59:18.839537 15865 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 09:59:18.839578 15865 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 09:59:18.839628 15865 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 09:59:18.849159 15865 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:35771
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35771
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 09:59:18.850299 15865 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 09:59:18.851617 15865 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 09:59:18.862772 15870 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:59:18.862707 15873 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:59:18.862746 15871 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 09:59:18.863073 15865 server_base.cc:1034] running on GCE node
I20260924 09:59:18.864022 15865 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 09:59:18.865283 15865 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 09:59:18.866499 15865 hybrid_clock.cc:648] HybridClock initialized: now 1790243958866441 us; error 30 us; skew 500 ppm
I20260924 09:59:18.866791 15865 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 09:59:18 dist-test-slave-w8d2 krb5kdc[15852](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243958, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 09:59:18.869660 15865 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260924 09:59:18.871038 15865 webserver.cc:460] Webserver started at http://127.0.97.62:40071/ using document root <none> and password file <none>
I20260924 09:59:18.871737 15865 fs_manager.cc:359] Metadata directory not provided
I20260924 09:59:18.871833 15865 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 09:59:18.872119 15865 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 09:59:18.874838 15865 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/minicluster-data/master-0/data/instance:
uuid: "6795636462c1442db8265257ea02ed3e"
format_stamp: "Formatted at 2026-09-24 09:59:18 on dist-test-slave-w8d2"
server_key: "ffcf085de30b7e372d06e0619e30efdc"
server_key_iv: "4fbe7fbf78fb40a8cae744590584df5c"
server_key_version: "encryptionkey@0"
I20260924 09:59:18.875589 15865 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/minicluster-data/master-0/wal/instance:
uuid: "6795636462c1442db8265257ea02ed3e"
format_stamp: "Formatted at 2026-09-24 09:59:18 on dist-test-slave-w8d2"
server_key: "ffcf085de30b7e372d06e0619e30efdc"
server_key_iv: "4fbe7fbf78fb40a8cae744590584df5c"
server_key_version: "encryptionkey@0"
I20260924 09:59:18.880208 15865 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260924 09:59:18.883266 15880 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 09:59:18.884465 15865 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260924 09:59:18.884603 15865 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/minicluster-data/master-0/wal
uuid: "6795636462c1442db8265257ea02ed3e"
format_stamp: "Formatted at 2026-09-24 09:59:18 on dist-test-slave-w8d2"
server_key: "ffcf085de30b7e372d06e0619e30efdc"
server_key_iv: "4fbe7fbf78fb40a8cae744590584df5c"
server_key_version: "encryptionkey@0"
I20260924 09:59:18.884783 15865 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 09:59:18.902969 15865 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 09:59:18.906682 15865 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 09:59:18.906911 15865 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 09:59:18.923383 15865 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:35771
I20260924 09:59:18.923393 15931 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:35771 every 8 connection(s)
I20260924 09:59:18.924647 15865 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/minicluster-data/master-0/data/info.pb
I20260924 09:59:18.928545 15932 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 09:59:18.931906 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu as pid 15865
I20260924 09:59:18.932092 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/minicluster-data/master-0/wal/instance
I20260924 09:59:18.932691 388 external_mini_cluster.cc:1428] Setting key d5e52277c921541d072cca4bb41ac5f6
I20260924 09:59:18.941076 15932 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6795636462c1442db8265257ea02ed3e: Bootstrap starting.
I20260924 09:59:18.944497 15932 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6795636462c1442db8265257ea02ed3e: Neither blocks nor log segments found. Creating new log.
I20260924 09:59:18.945443 15932 log.cc:824] T 00000000000000000000000000000000 P 6795636462c1442db8265257ea02ed3e: Log is configured to *not* fsync() on all Append() calls
Sep 24 09:59:18 dist-test-slave-w8d2 krb5kdc[15852](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243958, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 09:59:18.948351 15932 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6795636462c1442db8265257ea02ed3e: No bootstrap required, opened a new log
I20260924 09:59:18.954339 15935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:59:18.934429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57472 (local address 127.0.97.62:35771)
0924 09:59:18.935198 (+ 769us) server_negotiation.cc:206] Beginning negotiation
0924 09:59:18.935212 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 09:59:18.935288 (+ 76us) server_negotiation.cc:407] Connection header received
0924 09:59:18.936982 (+ 1694us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:59:18.937010 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:59:18.937531 (+ 521us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:59:18.937972 (+ 441us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:59:18.939150 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:59:18.941037 (+ 1887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:59:18.942370 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:59:18.943415 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:59:18.943583 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:59:18.947616 (+ 4033us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:59:18.947623 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:59:18.947637 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:59:18.947670 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:59:18.950518 (+ 2848us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:59:18.951352 (+ 834us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:59:18.951362 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:59:18.951367 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:59:18.951527 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:59:18.951919 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:59:18.951921 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:59:18.951923 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:59:18.952394 (+ 471us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:59:18.952602 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0924 09:59:18.953030 (+ 428us) server_negotiation.cc:299] Negotiation successful
0924 09:59:18.953344 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":534,"thread_start_us":313,"threads_started":1}
I20260924 09:59:18.956768 15932 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6795636462c1442db8265257ea02ed3e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6795636462c1442db8265257ea02ed3e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35771 } }
I20260924 09:59:18.957185 15932 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6795636462c1442db8265257ea02ed3e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 09:59:18.957278 15932 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6795636462c1442db8265257ea02ed3e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6795636462c1442db8265257ea02ed3e, State: Initialized, Role: FOLLOWER
I20260924 09:59:18.957796 15932 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6795636462c1442db8265257ea02ed3e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6795636462c1442db8265257ea02ed3e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35771 } }
I20260924 09:59:18.958003 15932 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6795636462c1442db8265257ea02ed3e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 09:59:18.958122 15932 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6795636462c1442db8265257ea02ed3e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 09:59:18.958256 15932 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6795636462c1442db8265257ea02ed3e [term 0 FOLLOWER]: Advancing to term 1
I20260924 09:59:18.960379 15932 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6795636462c1442db8265257ea02ed3e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6795636462c1442db8265257ea02ed3e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35771 } }
I20260924 09:59:18.960820 15932 leader_election.cc:302] T 00000000000000000000000000000000 P 6795636462c1442db8265257ea02ed3e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6795636462c1442db8265257ea02ed3e; no voters:
I20260924 09:59:18.961335 15932 leader_election.cc:288] T 00000000000000000000000000000000 P 6795636462c1442db8265257ea02ed3e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 09:59:18.961421 15937 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6795636462c1442db8265257ea02ed3e [term 1 FOLLOWER]: Leader election won for term 1
I20260924 09:59:18.962514 15937 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6795636462c1442db8265257ea02ed3e [term 1 LEADER]: Becoming Leader. State: Replica: 6795636462c1442db8265257ea02ed3e, State: Running, Role: LEADER
I20260924 09:59:18.962987 15937 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6795636462c1442db8265257ea02ed3e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6795636462c1442db8265257ea02ed3e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35771 } }
I20260924 09:59:18.963337 15932 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6795636462c1442db8265257ea02ed3e [sys.catalog]: configured and running, proceeding with master startup.
I20260924 09:59:18.965544 15938 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6795636462c1442db8265257ea02ed3e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6795636462c1442db8265257ea02ed3e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6795636462c1442db8265257ea02ed3e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35771 } } }
I20260924 09:59:18.965912 15938 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6795636462c1442db8265257ea02ed3e [sys.catalog]: This master's current role is: LEADER
I20260924 09:59:18.966360 15939 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6795636462c1442db8265257ea02ed3e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6795636462c1442db8265257ea02ed3e. Latest consensus state: current_term: 1 leader_uuid: "6795636462c1442db8265257ea02ed3e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6795636462c1442db8265257ea02ed3e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35771 } } }
I20260924 09:59:18.966658 15939 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6795636462c1442db8265257ea02ed3e [sys.catalog]: This master's current role is: LEADER
I20260924 09:59:18.968562 15944 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 09:59:18.973315 15944 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 09:59:18.982031 15944 catalog_manager.cc:1347] Generated new cluster ID: cd7c8363bac24ed7aa68f8264f131800
I20260924 09:59:18.982139 15944 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 09:59:18.989332 15944 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 09:59:18.990573 15944 catalog_manager.cc:1504] Loading token signing keys...
I20260924 09:59:19.000403 15944 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6795636462c1442db8265257ea02ed3e: Generated new TSK 0
I20260924 09:59:19.001143 15944 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260924 09:59:19.059165 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
/tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35771
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 09:59:19.366786 15958 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 09:59:19.367067 15958 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 09:59:19.367199 15958 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 09:59:19.374589 15958 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 09:59:19.374698 15958 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 09:59:19.374846 15958 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260924 09:59:19.384507 15958 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35771
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 09:59:19.385776 15958 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 09:59:19.387072 15958 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 09:59:19.399307 15963 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:59:19.399432 15966 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:59:19.399367 15964 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 09:59:19.400537 15958 server_base.cc:1034] running on GCE node
I20260924 09:59:19.401103 15958 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 09:59:19.401925 15958 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 09:59:19.403110 15958 hybrid_clock.cc:648] HybridClock initialized: now 1790243959403069 us; error 31 us; skew 500 ppm
I20260924 09:59:19.403410 15958 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 09:59:19 dist-test-slave-w8d2 krb5kdc[15852](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243959, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 09:59:19.411780 15958 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260924 09:59:19.413229 15958 webserver.cc:460] Webserver started at http://127.0.97.1:44043/ using document root <none> and password file <none>
I20260924 09:59:19.413887 15958 fs_manager.cc:359] Metadata directory not provided
I20260924 09:59:19.414003 15958 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 09:59:19.414283 15958 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 09:59:19.417135 15958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/minicluster-data/ts-0/data/instance:
uuid: "745a6616fac446f39b8f5ee7238814c0"
format_stamp: "Formatted at 2026-09-24 09:59:19 on dist-test-slave-w8d2"
server_key: "c3fb61bd489884f59220a26435b8283e"
server_key_iv: "d4548006e18d18dadb99a6ed0dd10a78"
server_key_version: "encryptionkey@0"
I20260924 09:59:19.417853 15958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/minicluster-data/ts-0/wal/instance:
uuid: "745a6616fac446f39b8f5ee7238814c0"
format_stamp: "Formatted at 2026-09-24 09:59:19 on dist-test-slave-w8d2"
server_key: "c3fb61bd489884f59220a26435b8283e"
server_key_iv: "d4548006e18d18dadb99a6ed0dd10a78"
server_key_version: "encryptionkey@0"
I20260924 09:59:19.422730 15958 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260924 09:59:19.426002 15973 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 09:59:19.427207 15958 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260924 09:59:19.427368 15958 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/minicluster-data/ts-0/wal
uuid: "745a6616fac446f39b8f5ee7238814c0"
format_stamp: "Formatted at 2026-09-24 09:59:19 on dist-test-slave-w8d2"
server_key: "c3fb61bd489884f59220a26435b8283e"
server_key_iv: "d4548006e18d18dadb99a6ed0dd10a78"
server_key_version: "encryptionkey@0"
I20260924 09:59:19.427546 15958 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 09:59:19.451047 15958 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 09:59:19.454767 15958 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 09:59:19.455032 15958 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 09:59:19.455950 15958 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 09:59:19.457201 15958 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 09:59:19.457274 15958 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 09:59:19.457398 15958 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 09:59:19.457448 15958 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 09:59:19.486176 15958 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:45337
I20260924 09:59:19.486209 16085 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:45337 every 8 connection(s)
I20260924 09:59:19.487473 15958 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/minicluster-data/ts-0/data/info.pb
I20260924 09:59:19.490049 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu as pid 15958
I20260924 09:59:19.490239 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/minicluster-data/ts-0/wal/instance
I20260924 09:59:19.490875 388 external_mini_cluster.cc:1428] Setting key e9d14b9762b2aedfb80a884e1f920214
Sep 24 09:59:19 dist-test-slave-w8d2 krb5kdc[15852](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243959, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 09:59:19.510006 16088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:59:19.489983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:33925 (local address 127.0.97.62:35771)
0924 09:59:19.491765 (+ 1782us) server_negotiation.cc:206] Beginning negotiation
0924 09:59:19.491771 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 09:59:19.492413 (+ 642us) server_negotiation.cc:407] Connection header received
0924 09:59:19.494779 (+ 2366us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:59:19.494783 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:59:19.494848 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:59:19.494977 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:59:19.497073 (+ 2096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:59:19.497782 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:59:19.500012 (+ 2230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:59:19.500429 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:59:19.500580 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:59:19.504463 (+ 3883us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:59:19.504468 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:59:19.504471 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:59:19.504505 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:59:19.506955 (+ 2450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:59:19.507823 (+ 868us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:59:19.507827 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:59:19.507829 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:59:19.507977 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:59:19.508400 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:59:19.508404 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:59:19.508406 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:59:19.508678 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:59:19.508788 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0924 09:59:19.509571 (+ 783us) server_negotiation.cc:299] Negotiation successful
0924 09:59:19.509736 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1657,"thread_start_us":1550,"threads_started":1}
I20260924 09:59:19.511437 16089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:59:19.490620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35771 (local address 127.0.97.1:33925)
0924 09:59:19.492262 (+ 1642us) negotiation.cc:107] Waiting for socket to connect
0924 09:59:19.492304 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 09:59:19.494462 (+ 2158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:59:19.495256 (+ 794us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:59:19.495268 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:59:19.495963 (+ 695us) client_negotiation.cc:189] Negotiated authn=SASL
0924 09:59:19.496835 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:59:19.496860 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:59:19.497949 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:59:19.497955 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:59:19.499858 (+ 1903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:59:19.499872 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:59:19.500651 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:59:19.500657 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:59:19.500932 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:59:19.501668 (+ 736us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 09:59:19.501692 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 09:59:19.504205 (+ 2513us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 09:59:19.507100 (+ 2895us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 09:59:19.507108 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 09:59:19.507125 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 09:59:19.507708 (+ 583us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 09:59:19.508070 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 09:59:19.508074 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 09:59:19.508077 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 09:59:19.508289 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 09:59:19.508794 (+ 505us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 09:59:19.508802 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 09:59:19.509188 (+ 386us) client_negotiation.cc:769] Sending connection context
0924 09:59:19.509485 (+ 297us) client_negotiation.cc:240] Negotiation successful
0924 09:59:19.510011 (+ 526us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1404,"thread_start_us":1194,"threads_started":1}
I20260924 09:59:19.513024 16086 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35771
I20260924 09:59:19.513334 16086 heartbeater.cc:464] Registering TS with master...
I20260924 09:59:19.514020 16086 heartbeater.cc:511] Master 127.0.97.62:35771 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260924 09:59:19.516057 15897 ts_manager.cc:194] Registered new tserver with Master: 745a6616fac446f39b8f5ee7238814c0 (127.0.97.1:45337)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260924 09:59:19.517561 15897 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:33925
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260924 09:59:19.540143 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
/tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35771
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 09:59:19.842793 16092 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 09:59:19.843086 16092 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 09:59:19.843221 16092 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 09:59:19.850325 16092 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 09:59:19.850430 16092 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 09:59:19.850538 16092 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260924 09:59:19.860064 16092 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35771
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 09:59:19.861215 16092 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 09:59:19.862401 16092 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 09:59:19.874511 16098 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:59:19.874574 16097 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:59:19.874606 16100 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 09:59:19.875391 16092 server_base.cc:1034] running on GCE node
I20260924 09:59:19.875972 16092 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 09:59:19.876819 16092 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 09:59:19.878027 16092 hybrid_clock.cc:648] HybridClock initialized: now 1790243959877953 us; error 47 us; skew 500 ppm
I20260924 09:59:19.878305 16092 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 09:59:19 dist-test-slave-w8d2 krb5kdc[15852](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243959, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 09:59:19.881101 16092 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260924 09:59:19.882403 16092 webserver.cc:460] Webserver started at http://127.0.97.2:45649/ using document root <none> and password file <none>
I20260924 09:59:19.882994 16092 fs_manager.cc:359] Metadata directory not provided
I20260924 09:59:19.883111 16092 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 09:59:19.883419 16092 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 09:59:19.886096 16092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/minicluster-data/ts-1/data/instance:
uuid: "e785aaa5a6044212b18b8397ce8483e0"
format_stamp: "Formatted at 2026-09-24 09:59:19 on dist-test-slave-w8d2"
server_key: "0fe5933f754a5200bd6a18cbad61fa62"
server_key_iv: "7b4dd5fbe13dbc0c63f092cfab0b95ef"
server_key_version: "encryptionkey@0"
I20260924 09:59:19.886765 16092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/minicluster-data/ts-1/wal/instance:
uuid: "e785aaa5a6044212b18b8397ce8483e0"
format_stamp: "Formatted at 2026-09-24 09:59:19 on dist-test-slave-w8d2"
server_key: "0fe5933f754a5200bd6a18cbad61fa62"
server_key_iv: "7b4dd5fbe13dbc0c63f092cfab0b95ef"
server_key_version: "encryptionkey@0"
I20260924 09:59:19.891616 16092 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260924 09:59:19.894972 16107 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 09:59:19.896292 16092 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 09:59:19.896437 16092 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/minicluster-data/ts-1/wal
uuid: "e785aaa5a6044212b18b8397ce8483e0"
format_stamp: "Formatted at 2026-09-24 09:59:19 on dist-test-slave-w8d2"
server_key: "0fe5933f754a5200bd6a18cbad61fa62"
server_key_iv: "7b4dd5fbe13dbc0c63f092cfab0b95ef"
server_key_version: "encryptionkey@0"
I20260924 09:59:19.896607 16092 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 09:59:19.935683 16092 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 09:59:19.939893 16092 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 09:59:19.940171 16092 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 09:59:19.941069 16092 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 09:59:19.942333 16092 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 09:59:19.942405 16092 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 09:59:19.942517 16092 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 09:59:19.942569 16092 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 09:59:19.971045 16092 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:40543
I20260924 09:59:19.971103 16219 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:40543 every 8 connection(s)
I20260924 09:59:19.972281 16092 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/minicluster-data/ts-1/data/info.pb
I20260924 09:59:19.979027 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu as pid 16092
I20260924 09:59:19.979189 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/minicluster-data/ts-1/wal/instance
I20260924 09:59:19.979643 388 external_mini_cluster.cc:1428] Setting key 25cfb9155f60782a974032e1874bd048
Sep 24 09:59:19 dist-test-slave-w8d2 krb5kdc[15852](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243959, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 09:59:19.992141 16088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:59:19.974659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:54343 (local address 127.0.97.62:35771)
0924 09:59:19.974835 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0924 09:59:19.974840 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 09:59:19.975909 (+ 1069us) server_negotiation.cc:407] Connection header received
0924 09:59:19.977642 (+ 1733us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:59:19.977647 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:59:19.977728 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:59:19.977814 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:59:19.979279 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:59:19.980065 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:59:19.982034 (+ 1969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:59:19.982620 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:59:19.982731 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:59:19.986113 (+ 3382us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:59:19.986119 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:59:19.986124 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:59:19.986160 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:59:19.989267 (+ 3107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:59:19.990098 (+ 831us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:59:19.990101 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:59:19.990104 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:59:19.990224 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:59:19.990632 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:59:19.990635 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:59:19.990637 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:59:19.990878 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:59:19.990989 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0924 09:59:19.991747 (+ 758us) server_negotiation.cc:299] Negotiation successful
0924 09:59:19.991909 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 09:59:19.993602 16222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:59:19.975079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35771 (local address 127.0.97.2:54343)
0924 09:59:19.975759 (+ 680us) negotiation.cc:107] Waiting for socket to connect
0924 09:59:19.975801 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 09:59:19.977387 (+ 1586us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:59:19.978041 (+ 654us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:59:19.978049 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:59:19.978528 (+ 479us) client_negotiation.cc:189] Negotiated authn=SASL
0924 09:59:19.979070 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:59:19.979089 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:59:19.980272 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:59:19.980277 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:59:19.981899 (+ 1622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:59:19.981911 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:59:19.982808 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:59:19.982813 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:59:19.983014 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:59:19.983640 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 09:59:19.983662 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 09:59:19.985866 (+ 2204us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 09:59:19.989389 (+ 3523us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 09:59:19.989398 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 09:59:19.989431 (+ 33us) client_negotiation.cc:760] calling sasl_client_step()
0924 09:59:19.989996 (+ 565us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 09:59:19.990317 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 09:59:19.990322 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 09:59:19.990326 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 09:59:19.990550 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 09:59:19.990986 (+ 436us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 09:59:19.990994 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 09:59:19.991399 (+ 405us) client_negotiation.cc:769] Sending connection context
0924 09:59:19.991689 (+ 290us) client_negotiation.cc:240] Negotiation successful
0924 09:59:19.992184 (+ 495us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":286,"threads_started":1}
I20260924 09:59:19.994983 16220 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35771
I20260924 09:59:19.995329 16220 heartbeater.cc:464] Registering TS with master...
I20260924 09:59:19.995998 16220 heartbeater.cc:511] Master 127.0.97.62:35771 requested a full tablet report, sending...
I20260924 09:59:19.997331 15897 ts_manager.cc:194] Registered new tserver with Master: e785aaa5a6044212b18b8397ce8483e0 (127.0.97.2:40543)
I20260924 09:59:19.998065 15897 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:54343
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260924 09:59:20.027674 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
/tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35771
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 09:59:20.329850 16225 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 09:59:20.330133 16225 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 09:59:20.330264 16225 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 09:59:20.337391 16225 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 09:59:20.337517 16225 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 09:59:20.337672 16225 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260924 09:59:20.347275 16225 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35771
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 09:59:20.348505 16225 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 09:59:20.349798 16225 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 09:59:20.362296 16230 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:59:20.362260 16233 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:59:20.362223 16231 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 09:59:20.363016 16225 server_base.cc:1034] running on GCE node
I20260924 09:59:20.363680 16225 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 09:59:20.364578 16225 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 09:59:20.365772 16225 hybrid_clock.cc:648] HybridClock initialized: now 1790243960365715 us; error 49 us; skew 500 ppm
I20260924 09:59:20.366070 16225 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 09:59:20 dist-test-slave-w8d2 krb5kdc[15852](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243960, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 09:59:20.369088 16225 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260924 09:59:20.370464 16225 webserver.cc:460] Webserver started at http://127.0.97.3:46083/ using document root <none> and password file <none>
I20260924 09:59:20.371135 16225 fs_manager.cc:359] Metadata directory not provided
I20260924 09:59:20.371266 16225 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 09:59:20.371556 16225 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 09:59:20.374303 16225 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/minicluster-data/ts-2/data/instance:
uuid: "f3295c0ca2c54aea8a83fe6cbfb8bc14"
format_stamp: "Formatted at 2026-09-24 09:59:20 on dist-test-slave-w8d2"
server_key: "336612d680ec5c24a7155b536bfeebd2"
server_key_iv: "999a87a783ef1800aab3eeb7bb26335f"
server_key_version: "encryptionkey@0"
I20260924 09:59:20.374996 16225 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/minicluster-data/ts-2/wal/instance:
uuid: "f3295c0ca2c54aea8a83fe6cbfb8bc14"
format_stamp: "Formatted at 2026-09-24 09:59:20 on dist-test-slave-w8d2"
server_key: "336612d680ec5c24a7155b536bfeebd2"
server_key_iv: "999a87a783ef1800aab3eeb7bb26335f"
server_key_version: "encryptionkey@0"
I20260924 09:59:20.379730 16225 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260924 09:59:20.382807 16240 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 09:59:20.384047 16225 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 09:59:20.384188 16225 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/minicluster-data/ts-2/wal
uuid: "f3295c0ca2c54aea8a83fe6cbfb8bc14"
format_stamp: "Formatted at 2026-09-24 09:59:20 on dist-test-slave-w8d2"
server_key: "336612d680ec5c24a7155b536bfeebd2"
server_key_iv: "999a87a783ef1800aab3eeb7bb26335f"
server_key_version: "encryptionkey@0"
I20260924 09:59:20.384349 16225 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 09:59:20.402204 16225 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 09:59:20.405686 16225 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 09:59:20.405932 16225 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 09:59:20.406781 16225 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 09:59:20.408022 16225 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 09:59:20.408095 16225 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 09:59:20.408208 16225 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 09:59:20.408258 16225 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 09:59:20.436870 16225 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:39627
I20260924 09:59:20.436930 16352 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:39627 every 8 connection(s)
I20260924 09:59:20.438038 16225 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/minicluster-data/ts-2/data/info.pb
I20260924 09:59:20.445989 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu as pid 16225
I20260924 09:59:20.446162 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/minicluster-data/ts-2/wal/instance
I20260924 09:59:20.446575 388 external_mini_cluster.cc:1428] Setting key 194c38fcaac6760e8d3f717941d4c1f8
Sep 24 09:59:20 dist-test-slave-w8d2 krb5kdc[15852](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243960, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 09:59:20.455878 16088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:59:20.440388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:55265 (local address 127.0.97.62:35771)
0924 09:59:20.440553 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0924 09:59:20.440558 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 09:59:20.441620 (+ 1062us) server_negotiation.cc:407] Connection header received
0924 09:59:20.443345 (+ 1725us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:59:20.443352 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:59:20.443433 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:59:20.443520 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:59:20.444880 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:59:20.445591 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:59:20.447121 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:59:20.447679 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:59:20.447784 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:59:20.450825 (+ 3041us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:59:20.450832 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:59:20.450836 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:59:20.450867 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:59:20.453182 (+ 2315us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:59:20.453805 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:59:20.453811 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:59:20.453815 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:59:20.453936 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:59:20.454318 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:59:20.454323 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:59:20.454327 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:59:20.454583 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:59:20.454716 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0924 09:59:20.455343 (+ 627us) server_negotiation.cc:299] Negotiation successful
0924 09:59:20.455549 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 09:59:20.456705 16355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:59:20.440827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35771 (local address 127.0.97.3:55265)
0924 09:59:20.441480 (+ 653us) negotiation.cc:107] Waiting for socket to connect
0924 09:59:20.441515 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 09:59:20.443075 (+ 1560us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:59:20.443742 (+ 667us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:59:20.443750 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:59:20.444212 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0924 09:59:20.444727 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:59:20.444743 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:59:20.445730 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:59:20.445734 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:59:20.446995 (+ 1261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:59:20.447006 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:59:20.447877 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:59:20.447884 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:59:20.448103 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:59:20.448682 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 09:59:20.448706 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 09:59:20.450639 (+ 1933us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 09:59:20.453300 (+ 2661us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 09:59:20.453309 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 09:59:20.453323 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 09:59:20.453696 (+ 373us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 09:59:20.454040 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 09:59:20.454045 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 09:59:20.454049 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 09:59:20.454220 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 09:59:20.454693 (+ 473us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 09:59:20.454700 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 09:59:20.454988 (+ 288us) client_negotiation.cc:769] Sending connection context
0924 09:59:20.455220 (+ 232us) client_negotiation.cc:240] Negotiation successful
0924 09:59:20.455708 (+ 488us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":250,"threads_started":1}
I20260924 09:59:20.457974 16353 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35771
I20260924 09:59:20.458223 16353 heartbeater.cc:464] Registering TS with master...
I20260924 09:59:20.458846 16353 heartbeater.cc:511] Master 127.0.97.62:35771 requested a full tablet report, sending...
I20260924 09:59:20.460350 15897 ts_manager.cc:194] Registered new tserver with Master: f3295c0ca2c54aea8a83fe6cbfb8bc14 (127.0.97.3:39627)
I20260924 09:59:20.461247 15897 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:55265
I20260924 09:59:20.461408 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 09:59:20.477609 16088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:59:20.466390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57480 (local address 127.0.97.62:35771)
0924 09:59:20.466568 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0924 09:59:20.466575 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 09:59:20.466925 (+ 350us) server_negotiation.cc:407] Connection header received
0924 09:59:20.467050 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:59:20.467054 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:59:20.467110 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:59:20.467187 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:59:20.468433 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:59:20.469103 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:59:20.470445 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:59:20.470856 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:59:20.470942 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:59:20.473234 (+ 2292us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:59:20.473238 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:59:20.473241 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:59:20.473269 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:59:20.475573 (+ 2304us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:59:20.476139 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:59:20.476142 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:59:20.476144 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:59:20.476245 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:59:20.476511 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:59:20.476513 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:59:20.476515 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:59:20.476812 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:59:20.476977 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0924 09:59:20.477200 (+ 223us) server_negotiation.cc:299] Negotiation successful
0924 09:59:20.477354 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 09:59:20.481236 15897 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57480:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 09:59:20.488126 15897 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 09:59:20.508399 16366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:59:20.499348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35156 (local address 127.0.97.1:45337)
0924 09:59:20.499947 (+ 599us) server_negotiation.cc:206] Beginning negotiation
0924 09:59:20.499956 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 09:59:20.500602 (+ 646us) server_negotiation.cc:407] Connection header received
0924 09:59:20.500895 (+ 293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:59:20.500904 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:59:20.501198 (+ 294us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:59:20.501349 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 09:59:20.502710 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:59:20.504195 (+ 1485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:59:20.506086 (+ 1891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:59:20.507311 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:59:20.507426 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:59:20.507640 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0924 09:59:20.507917 (+ 277us) server_negotiation.cc:299] Negotiation successful
0924 09:59:20.508090 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":445,"thread_start_us":305,"threads_started":1}
I20260924 09:59:20.508481 16367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:59:20.499112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45337 (local address 127.0.0.1:35156)
0924 09:59:20.500434 (+ 1322us) negotiation.cc:107] Waiting for socket to connect
0924 09:59:20.500506 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0924 09:59:20.500679 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:59:20.501651 (+ 972us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:59:20.501658 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:59:20.501688 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 09:59:20.502520 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:59:20.502531 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:59:20.504324 (+ 1793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:59:20.504328 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:59:20.505888 (+ 1560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:59:20.505900 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:59:20.507489 (+ 1589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:59:20.507495 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:59:20.507683 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:59:20.507736 (+ 53us) client_negotiation.cc:769] Sending connection context
0924 09:59:20.507859 (+ 123us) client_negotiation.cc:240] Negotiation successful
0924 09:59:20.508027 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1070,"mutex_wait_us":26,"spinlock_wait_cycles":1280,"thread_start_us":992,"threads_started":1}
I20260924 09:59:20.509123 16365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:59:20.499643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:40543 (local address 127.0.0.1:48560)
0924 09:59:20.500434 (+ 791us) negotiation.cc:107] Waiting for socket to connect
0924 09:59:20.500502 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0924 09:59:20.500679 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:59:20.501788 (+ 1109us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:59:20.501792 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:59:20.501814 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 09:59:20.502185 (+ 371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:59:20.502199 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:59:20.503971 (+ 1772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:59:20.503976 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:59:20.506664 (+ 2688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:59:20.506679 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:59:20.508356 (+ 1677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:59:20.508362 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:59:20.508542 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:59:20.508578 (+ 36us) client_negotiation.cc:769] Sending connection context
0924 09:59:20.508639 (+ 61us) client_negotiation.cc:240] Negotiation successful
0924 09:59:20.508743 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":574,"mutex_wait_us":45,"spinlock_wait_cycles":10112,"thread_start_us":746,"threads_started":1}
I20260924 09:59:20.510113 16368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:59:20.500244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48560 (local address 127.0.97.2:40543)
0924 09:59:20.501255 (+ 1011us) server_negotiation.cc:206] Beginning negotiation
0924 09:59:20.501263 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 09:59:20.501290 (+ 27us) server_negotiation.cc:407] Connection header received
0924 09:59:20.501373 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:59:20.501379 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:59:20.501587 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:59:20.502036 (+ 449us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 09:59:20.502333 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:59:20.503860 (+ 1527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:59:20.506809 (+ 2949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:59:20.508121 (+ 1312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:59:20.509413 (+ 1292us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:59:20.509607 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0924 09:59:20.509668 (+ 61us) server_negotiation.cc:299] Negotiation successful
0924 09:59:20.509817 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":865,"thread_start_us":746,"threads_started":1}
I20260924 09:59:20.510655 16369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:59:20.500919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:39627 (local address 127.0.0.1:38710)
0924 09:59:20.503030 (+ 2111us) negotiation.cc:107] Waiting for socket to connect
0924 09:59:20.503046 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 09:59:20.503158 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:59:20.503732 (+ 574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:59:20.503736 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:59:20.503762 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 09:59:20.504136 (+ 374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:59:20.504147 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:59:20.506041 (+ 1894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:59:20.506046 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:59:20.508137 (+ 2091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:59:20.508151 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:59:20.509936 (+ 1785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:59:20.509942 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:59:20.510097 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:59:20.510120 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 09:59:20.510196 (+ 76us) client_negotiation.cc:240] Negotiation successful
0924 09:59:20.510307 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1983,"mutex_wait_us":1,"thread_start_us":1866,"threads_started":1}
I20260924 09:59:20.510694 16370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:59:20.501397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38710 (local address 127.0.97.3:39627)
0924 09:59:20.502035 (+ 638us) server_negotiation.cc:206] Beginning negotiation
0924 09:59:20.502043 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 09:59:20.503284 (+ 1241us) server_negotiation.cc:407] Connection header received
0924 09:59:20.503376 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:59:20.503383 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:59:20.503568 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:59:20.503723 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 09:59:20.504320 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:59:20.505872 (+ 1552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:59:20.508307 (+ 2435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:59:20.509493 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:59:20.509604 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:59:20.509820 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0924 09:59:20.510250 (+ 430us) server_negotiation.cc:299] Negotiation successful
0924 09:59:20.510394 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":480,"thread_start_us":345,"threads_started":1}
I20260924 09:59:20.512563 16021 tablet_service.cc:1467] Processing CreateTablet for tablet 7dfd7f95a5a54851998decd980a3d668 (DEFAULT_TABLE table=test-table [id=44537e0409154114afc862de6c6276cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 09:59:20.513321 16155 tablet_service.cc:1467] Processing CreateTablet for tablet 7dfd7f95a5a54851998decd980a3d668 (DEFAULT_TABLE table=test-table [id=44537e0409154114afc862de6c6276cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 09:59:20.513368 16288 tablet_service.cc:1467] Processing CreateTablet for tablet 7dfd7f95a5a54851998decd980a3d668 (DEFAULT_TABLE table=test-table [id=44537e0409154114afc862de6c6276cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 09:59:20.514042 16021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7dfd7f95a5a54851998decd980a3d668. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 09:59:20.514644 16155 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7dfd7f95a5a54851998decd980a3d668. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 09:59:20.514690 16288 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7dfd7f95a5a54851998decd980a3d668. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 09:59:20.520398 16086 heartbeater.cc:503] Master 127.0.97.62:35771 was elected leader, sending a full tablet report...
I20260924 09:59:20.525676 16371 tablet_bootstrap.cc:492] T 7dfd7f95a5a54851998decd980a3d668 P f3295c0ca2c54aea8a83fe6cbfb8bc14: Bootstrap starting.
I20260924 09:59:20.528591 16371 tablet_bootstrap.cc:654] T 7dfd7f95a5a54851998decd980a3d668 P f3295c0ca2c54aea8a83fe6cbfb8bc14: Neither blocks nor log segments found. Creating new log.
I20260924 09:59:20.529544 16371 log.cc:824] T 7dfd7f95a5a54851998decd980a3d668 P f3295c0ca2c54aea8a83fe6cbfb8bc14: Log is configured to *not* fsync() on all Append() calls
I20260924 09:59:20.532518 16372 tablet_bootstrap.cc:492] T 7dfd7f95a5a54851998decd980a3d668 P 745a6616fac446f39b8f5ee7238814c0: Bootstrap starting.
I20260924 09:59:20.533329 16373 tablet_bootstrap.cc:492] T 7dfd7f95a5a54851998decd980a3d668 P e785aaa5a6044212b18b8397ce8483e0: Bootstrap starting.
I20260924 09:59:20.536221 16372 tablet_bootstrap.cc:654] T 7dfd7f95a5a54851998decd980a3d668 P 745a6616fac446f39b8f5ee7238814c0: Neither blocks nor log segments found. Creating new log.
I20260924 09:59:20.536453 16373 tablet_bootstrap.cc:654] T 7dfd7f95a5a54851998decd980a3d668 P e785aaa5a6044212b18b8397ce8483e0: Neither blocks nor log segments found. Creating new log.
I20260924 09:59:20.537194 16372 log.cc:824] T 7dfd7f95a5a54851998decd980a3d668 P 745a6616fac446f39b8f5ee7238814c0: Log is configured to *not* fsync() on all Append() calls
I20260924 09:59:20.537609 16373 log.cc:824] T 7dfd7f95a5a54851998decd980a3d668 P e785aaa5a6044212b18b8397ce8483e0: Log is configured to *not* fsync() on all Append() calls
I20260924 09:59:20.537978 16371 tablet_bootstrap.cc:492] T 7dfd7f95a5a54851998decd980a3d668 P f3295c0ca2c54aea8a83fe6cbfb8bc14: No bootstrap required, opened a new log
I20260924 09:59:20.538278 16371 ts_tablet_manager.cc:1397] T 7dfd7f95a5a54851998decd980a3d668 P f3295c0ca2c54aea8a83fe6cbfb8bc14: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.003s
I20260924 09:59:20.539698 16372 tablet_bootstrap.cc:492] T 7dfd7f95a5a54851998decd980a3d668 P 745a6616fac446f39b8f5ee7238814c0: No bootstrap required, opened a new log
I20260924 09:59:20.539952 16372 ts_tablet_manager.cc:1397] T 7dfd7f95a5a54851998decd980a3d668 P 745a6616fac446f39b8f5ee7238814c0: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.004s
I20260924 09:59:20.541450 16373 tablet_bootstrap.cc:492] T 7dfd7f95a5a54851998decd980a3d668 P e785aaa5a6044212b18b8397ce8483e0: No bootstrap required, opened a new log
I20260924 09:59:20.541684 16373 ts_tablet_manager.cc:1397] T 7dfd7f95a5a54851998decd980a3d668 P e785aaa5a6044212b18b8397ce8483e0: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.000s
I20260924 09:59:20.545210 16371 raft_consensus.cc:348] T 7dfd7f95a5a54851998decd980a3d668 P f3295c0ca2c54aea8a83fe6cbfb8bc14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745a6616fac446f39b8f5ee7238814c0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45337 } } peers { permanent_uuid: "f3295c0ca2c54aea8a83fe6cbfb8bc14" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39627 } } peers { permanent_uuid: "e785aaa5a6044212b18b8397ce8483e0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40543 } }
I20260924 09:59:20.545498 16371 raft_consensus.cc:374] T 7dfd7f95a5a54851998decd980a3d668 P f3295c0ca2c54aea8a83fe6cbfb8bc14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 09:59:20.545553 16371 raft_consensus.cc:729] T 7dfd7f95a5a54851998decd980a3d668 P f3295c0ca2c54aea8a83fe6cbfb8bc14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3295c0ca2c54aea8a83fe6cbfb8bc14, State: Initialized, Role: FOLLOWER
I20260924 09:59:20.546121 16371 consensus_queue.cc:260] T 7dfd7f95a5a54851998decd980a3d668 P f3295c0ca2c54aea8a83fe6cbfb8bc14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745a6616fac446f39b8f5ee7238814c0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45337 } } peers { permanent_uuid: "f3295c0ca2c54aea8a83fe6cbfb8bc14" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39627 } } peers { permanent_uuid: "e785aaa5a6044212b18b8397ce8483e0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40543 } }
I20260924 09:59:20.547106 16353 heartbeater.cc:503] Master 127.0.97.62:35771 was elected leader, sending a full tablet report...
I20260924 09:59:20.547552 16371 ts_tablet_manager.cc:1428] T 7dfd7f95a5a54851998decd980a3d668 P f3295c0ca2c54aea8a83fe6cbfb8bc14: Time spent starting tablet: real 0.009s user 0.006s sys 0.005s
I20260924 09:59:20.551556 16373 raft_consensus.cc:348] T 7dfd7f95a5a54851998decd980a3d668 P e785aaa5a6044212b18b8397ce8483e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745a6616fac446f39b8f5ee7238814c0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45337 } } peers { permanent_uuid: "f3295c0ca2c54aea8a83fe6cbfb8bc14" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39627 } } peers { permanent_uuid: "e785aaa5a6044212b18b8397ce8483e0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40543 } }
I20260924 09:59:20.552029 16373 raft_consensus.cc:374] T 7dfd7f95a5a54851998decd980a3d668 P e785aaa5a6044212b18b8397ce8483e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 09:59:20.552131 16373 raft_consensus.cc:729] T 7dfd7f95a5a54851998decd980a3d668 P e785aaa5a6044212b18b8397ce8483e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e785aaa5a6044212b18b8397ce8483e0, State: Initialized, Role: FOLLOWER
I20260924 09:59:20.552186 16372 raft_consensus.cc:348] T 7dfd7f95a5a54851998decd980a3d668 P 745a6616fac446f39b8f5ee7238814c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745a6616fac446f39b8f5ee7238814c0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45337 } } peers { permanent_uuid: "f3295c0ca2c54aea8a83fe6cbfb8bc14" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39627 } } peers { permanent_uuid: "e785aaa5a6044212b18b8397ce8483e0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40543 } }
I20260924 09:59:20.552614 16372 raft_consensus.cc:374] T 7dfd7f95a5a54851998decd980a3d668 P 745a6616fac446f39b8f5ee7238814c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 09:59:20.552713 16372 raft_consensus.cc:729] T 7dfd7f95a5a54851998decd980a3d668 P 745a6616fac446f39b8f5ee7238814c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 745a6616fac446f39b8f5ee7238814c0, State: Initialized, Role: FOLLOWER
I20260924 09:59:20.552742 16373 consensus_queue.cc:260] T 7dfd7f95a5a54851998decd980a3d668 P e785aaa5a6044212b18b8397ce8483e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745a6616fac446f39b8f5ee7238814c0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45337 } } peers { permanent_uuid: "f3295c0ca2c54aea8a83fe6cbfb8bc14" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39627 } } peers { permanent_uuid: "e785aaa5a6044212b18b8397ce8483e0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40543 } }
I20260924 09:59:20.553411 16372 consensus_queue.cc:260] T 7dfd7f95a5a54851998decd980a3d668 P 745a6616fac446f39b8f5ee7238814c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745a6616fac446f39b8f5ee7238814c0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45337 } } peers { permanent_uuid: "f3295c0ca2c54aea8a83fe6cbfb8bc14" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39627 } } peers { permanent_uuid: "e785aaa5a6044212b18b8397ce8483e0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40543 } }
I20260924 09:59:20.553745 16220 heartbeater.cc:503] Master 127.0.97.62:35771 was elected leader, sending a full tablet report...
I20260924 09:59:20.554471 16373 ts_tablet_manager.cc:1428] T 7dfd7f95a5a54851998decd980a3d668 P e785aaa5a6044212b18b8397ce8483e0: Time spent starting tablet: real 0.013s user 0.009s sys 0.005s
I20260924 09:59:20.556401 16372 ts_tablet_manager.cc:1428] T 7dfd7f95a5a54851998decd980a3d668 P 745a6616fac446f39b8f5ee7238814c0: Time spent starting tablet: real 0.016s user 0.012s sys 0.001s
I20260924 09:59:20.562492 16379 raft_consensus.cc:482] T 7dfd7f95a5a54851998decd980a3d668 P 745a6616fac446f39b8f5ee7238814c0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 09:59:20.562705 16379 raft_consensus.cc:504] T 7dfd7f95a5a54851998decd980a3d668 P 745a6616fac446f39b8f5ee7238814c0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745a6616fac446f39b8f5ee7238814c0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45337 } } peers { permanent_uuid: "f3295c0ca2c54aea8a83fe6cbfb8bc14" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39627 } } peers { permanent_uuid: "e785aaa5a6044212b18b8397ce8483e0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40543 } }
I20260924 09:59:20.564468 16379 leader_election.cc:288] T 7dfd7f95a5a54851998decd980a3d668 P 745a6616fac446f39b8f5ee7238814c0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f3295c0ca2c54aea8a83fe6cbfb8bc14 (127.0.97.3:39627), e785aaa5a6044212b18b8397ce8483e0 (127.0.97.2:40543)
I20260924 09:59:20.571483 16380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:59:20.564820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:39627 (local address 127.0.97.1:53001)
0924 09:59:20.565627 (+ 807us) negotiation.cc:107] Waiting for socket to connect
0924 09:59:20.565648 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 09:59:20.565768 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:59:20.566064 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:59:20.566068 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:59:20.566093 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 09:59:20.566408 (+ 315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:59:20.566417 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:59:20.567388 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:59:20.567394 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:59:20.569281 (+ 1887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:59:20.569295 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:59:20.570844 (+ 1549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:59:20.570848 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:59:20.570968 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:59:20.570991 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 09:59:20.571061 (+ 70us) client_negotiation.cc:240] Negotiation successful
0924 09:59:20.571178 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":695,"thread_start_us":514,"threads_started":1}
I20260924 09:59:20.571492 16370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:59:20.564825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:53001 (local address 127.0.97.3:39627)
0924 09:59:20.565169 (+ 344us) server_negotiation.cc:206] Beginning negotiation
0924 09:59:20.565176 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 09:59:20.565723 (+ 547us) server_negotiation.cc:407] Connection header received
0924 09:59:20.565895 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:59:20.565899 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:59:20.565968 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:59:20.566041 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 09:59:20.566532 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:59:20.567270 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:59:20.569465 (+ 2195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:59:20.570646 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:59:20.570769 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:59:20.570874 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0924 09:59:20.571102 (+ 228us) server_negotiation.cc:299] Negotiation successful
0924 09:59:20.571252 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":192}
I20260924 09:59:20.572507 16308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dfd7f95a5a54851998decd980a3d668" candidate_uuid: "745a6616fac446f39b8f5ee7238814c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3295c0ca2c54aea8a83fe6cbfb8bc14" is_pre_election: true
I20260924 09:59:20.573055 16308 raft_consensus.cc:2454] T 7dfd7f95a5a54851998decd980a3d668 P f3295c0ca2c54aea8a83fe6cbfb8bc14 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 745a6616fac446f39b8f5ee7238814c0 in term 0.
I20260924 09:59:20.573884 16368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:59:20.564943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:57907 (local address 127.0.97.2:40543)
0924 09:59:20.565137 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0924 09:59:20.565144 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 09:59:20.567894 (+ 2750us) server_negotiation.cc:407] Connection header received
0924 09:59:20.568012 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:59:20.568017 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:59:20.568087 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:59:20.568186 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 09:59:20.568665 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:59:20.569586 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:59:20.571921 (+ 2335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:59:20.573040 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:59:20.573184 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:59:20.573269 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0924 09:59:20.573532 (+ 263us) server_negotiation.cc:299] Negotiation successful
0924 09:59:20.573651 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 09:59:20.573752 15975 leader_election.cc:302] T 7dfd7f95a5a54851998decd980a3d668 P 745a6616fac446f39b8f5ee7238814c0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 745a6616fac446f39b8f5ee7238814c0, f3295c0ca2c54aea8a83fe6cbfb8bc14; no voters:
I20260924 09:59:20.573937 16381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:59:20.564852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:40543 (local address 127.0.97.1:57907)
0924 09:59:20.567790 (+ 2938us) negotiation.cc:107] Waiting for socket to connect
0924 09:59:20.567809 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 09:59:20.567916 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:59:20.568189 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:59:20.568192 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:59:20.568210 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 09:59:20.568524 (+ 314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:59:20.568532 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:59:20.569721 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:59:20.569726 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:59:20.571776 (+ 2050us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:59:20.571791 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:59:20.573236 (+ 1445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:59:20.573243 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:59:20.573387 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:59:20.573419 (+ 32us) client_negotiation.cc:769] Sending connection context
0924 09:59:20.573503 (+ 84us) client_negotiation.cc:240] Negotiation successful
0924 09:59:20.573630 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2803,"thread_start_us":2576,"threads_started":1}
I20260924 09:59:20.574215 16379 raft_consensus.cc:2781] T 7dfd7f95a5a54851998decd980a3d668 P 745a6616fac446f39b8f5ee7238814c0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 09:59:20.574354 16379 raft_consensus.cc:482] T 7dfd7f95a5a54851998decd980a3d668 P 745a6616fac446f39b8f5ee7238814c0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 09:59:20.574448 16379 raft_consensus.cc:3037] T 7dfd7f95a5a54851998decd980a3d668 P 745a6616fac446f39b8f5ee7238814c0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 09:59:20.574788 16175 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dfd7f95a5a54851998decd980a3d668" candidate_uuid: "745a6616fac446f39b8f5ee7238814c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e785aaa5a6044212b18b8397ce8483e0" is_pre_election: true
I20260924 09:59:20.575205 16175 raft_consensus.cc:2454] T 7dfd7f95a5a54851998decd980a3d668 P e785aaa5a6044212b18b8397ce8483e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 745a6616fac446f39b8f5ee7238814c0 in term 0.
I20260924 09:59:20.576771 16379 raft_consensus.cc:504] T 7dfd7f95a5a54851998decd980a3d668 P 745a6616fac446f39b8f5ee7238814c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745a6616fac446f39b8f5ee7238814c0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45337 } } peers { permanent_uuid: "f3295c0ca2c54aea8a83fe6cbfb8bc14" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39627 } } peers { permanent_uuid: "e785aaa5a6044212b18b8397ce8483e0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40543 } }
I20260924 09:59:20.577298 16379 leader_election.cc:288] T 7dfd7f95a5a54851998decd980a3d668 P 745a6616fac446f39b8f5ee7238814c0 [CANDIDATE]: Term 1 election: Requested vote from peers f3295c0ca2c54aea8a83fe6cbfb8bc14 (127.0.97.3:39627), e785aaa5a6044212b18b8397ce8483e0 (127.0.97.2:40543)
I20260924 09:59:20.577797 16175 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dfd7f95a5a54851998decd980a3d668" candidate_uuid: "745a6616fac446f39b8f5ee7238814c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e785aaa5a6044212b18b8397ce8483e0"
I20260924 09:59:20.577797 16308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dfd7f95a5a54851998decd980a3d668" candidate_uuid: "745a6616fac446f39b8f5ee7238814c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3295c0ca2c54aea8a83fe6cbfb8bc14"
I20260924 09:59:20.577992 16175 raft_consensus.cc:3037] T 7dfd7f95a5a54851998decd980a3d668 P e785aaa5a6044212b18b8397ce8483e0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 09:59:20.577989 16308 raft_consensus.cc:3037] T 7dfd7f95a5a54851998decd980a3d668 P f3295c0ca2c54aea8a83fe6cbfb8bc14 [term 0 FOLLOWER]: Advancing to term 1
I20260924 09:59:20.581043 16175 raft_consensus.cc:2454] T 7dfd7f95a5a54851998decd980a3d668 P e785aaa5a6044212b18b8397ce8483e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 745a6616fac446f39b8f5ee7238814c0 in term 1.
I20260924 09:59:20.581055 16308 raft_consensus.cc:2454] T 7dfd7f95a5a54851998decd980a3d668 P f3295c0ca2c54aea8a83fe6cbfb8bc14 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 745a6616fac446f39b8f5ee7238814c0 in term 1.
I20260924 09:59:20.581424 15977 leader_election.cc:302] T 7dfd7f95a5a54851998decd980a3d668 P 745a6616fac446f39b8f5ee7238814c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 745a6616fac446f39b8f5ee7238814c0, e785aaa5a6044212b18b8397ce8483e0; no voters:
I20260924 09:59:20.581697 16379 raft_consensus.cc:2781] T 7dfd7f95a5a54851998decd980a3d668 P 745a6616fac446f39b8f5ee7238814c0 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 09:59:20.582069 16379 raft_consensus.cc:686] T 7dfd7f95a5a54851998decd980a3d668 P 745a6616fac446f39b8f5ee7238814c0 [term 1 LEADER]: Becoming Leader. State: Replica: 745a6616fac446f39b8f5ee7238814c0, State: Running, Role: LEADER
I20260924 09:59:20.582572 16379 consensus_queue.cc:237] T 7dfd7f95a5a54851998decd980a3d668 P 745a6616fac446f39b8f5ee7238814c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745a6616fac446f39b8f5ee7238814c0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45337 } } peers { permanent_uuid: "f3295c0ca2c54aea8a83fe6cbfb8bc14" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39627 } } peers { permanent_uuid: "e785aaa5a6044212b18b8397ce8483e0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40543 } }
I20260924 09:59:20.586751 15897 catalog_manager.cc:5495] T 7dfd7f95a5a54851998decd980a3d668 P 745a6616fac446f39b8f5ee7238814c0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 745a6616fac446f39b8f5ee7238814c0 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "745a6616fac446f39b8f5ee7238814c0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745a6616fac446f39b8f5ee7238814c0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45337 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f3295c0ca2c54aea8a83fe6cbfb8bc14" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39627 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e785aaa5a6044212b18b8397ce8483e0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40543 } health_report { overall_health: UNKNOWN } } }
I20260924 09:59:20.609905 16366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:59:20.604844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35158 (local address 127.0.97.1:45337)
0924 09:59:20.605002 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0924 09:59:20.605006 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 09:59:20.605103 (+ 97us) server_negotiation.cc:407] Connection header received
0924 09:59:20.605269 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:59:20.605272 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:59:20.605363 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:59:20.605479 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 09:59:20.605772 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:59:20.606480 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:59:20.608072 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:59:20.608450 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:59:20.608537 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:59:20.608966 (+ 429us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 09:59:20.609441 (+ 475us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 09:59:20.609543 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0924 09:59:20.609635 (+ 92us) server_negotiation.cc:299] Negotiation successful
0924 09:59:20.609709 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 09:59:20.617486 16308 raft_consensus.cc:1261] T 7dfd7f95a5a54851998decd980a3d668 P f3295c0ca2c54aea8a83fe6cbfb8bc14 [term 1 FOLLOWER]: Refusing update from remote peer 745a6616fac446f39b8f5ee7238814c0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 09:59:20.617481 16175 raft_consensus.cc:1261] T 7dfd7f95a5a54851998decd980a3d668 P e785aaa5a6044212b18b8397ce8483e0 [term 1 FOLLOWER]: Refusing update from remote peer 745a6616fac446f39b8f5ee7238814c0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 09:59:20.618355 16382 consensus_queue.cc:1035] T 7dfd7f95a5a54851998decd980a3d668 P 745a6616fac446f39b8f5ee7238814c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e785aaa5a6044212b18b8397ce8483e0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40543 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 09:59:20.618678 16379 consensus_queue.cc:1035] T 7dfd7f95a5a54851998decd980a3d668 P 745a6616fac446f39b8f5ee7238814c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f3295c0ca2c54aea8a83fe6cbfb8bc14" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39627 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 09:59:20.631788 16386 mvcc.cc:205] Tried to move back new op lower bound from 7332839262680330240 to 7332839262549962752. Current Snapshot: MvccSnapshot[applied={T|T < 7332839262680330240}]
I20260924 09:59:20.640636 15897 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57480:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 09:59:20.640861 15897 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57480:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 09:59:20.644057 15897 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6795636462c1442db8265257ea02ed3e: Sending DeleteTablet for 3 replicas of tablet 7dfd7f95a5a54851998decd980a3d668
I20260924 09:59:20.645056 16021 tablet_service.cc:1514] Processing DeleteTablet for tablet 7dfd7f95a5a54851998decd980a3d668 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 09:59:20 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:35156
I20260924 09:59:20.645129 16288 tablet_service.cc:1514] Processing DeleteTablet for tablet 7dfd7f95a5a54851998decd980a3d668 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 09:59:20 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:38710
I20260924 09:59:20.645191 16155 tablet_service.cc:1514] Processing DeleteTablet for tablet 7dfd7f95a5a54851998decd980a3d668 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 09:59:20 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:48560
I20260924 09:59:20.645401 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu pid 15958
I20260924 09:59:20.657454 16395 tablet_replica.cc:330] T 7dfd7f95a5a54851998decd980a3d668 P 745a6616fac446f39b8f5ee7238814c0: stopping tablet replica
I20260924 09:59:20.658034 16397 tablet_replica.cc:330] T 7dfd7f95a5a54851998decd980a3d668 P e785aaa5a6044212b18b8397ce8483e0: stopping tablet replica
I20260924 09:59:20.658656 16397 raft_consensus.cc:2229] T 7dfd7f95a5a54851998decd980a3d668 P e785aaa5a6044212b18b8397ce8483e0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 09:59:20.659183 16395 raft_consensus.cc:2229] T 7dfd7f95a5a54851998decd980a3d668 P 745a6616fac446f39b8f5ee7238814c0 [term 1 LEADER]: Raft consensus shutting down.
I20260924 09:59:20.659966 16395 raft_consensus.cc:2258] T 7dfd7f95a5a54851998decd980a3d668 P 745a6616fac446f39b8f5ee7238814c0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 09:59:20.660526 16397 raft_consensus.cc:2258] T 7dfd7f95a5a54851998decd980a3d668 P e785aaa5a6044212b18b8397ce8483e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 09:59:20.663497 16396 tablet_replica.cc:330] T 7dfd7f95a5a54851998decd980a3d668 P f3295c0ca2c54aea8a83fe6cbfb8bc14: stopping tablet replica
I20260924 09:59:20.663623 16395 ts_tablet_manager.cc:1905] T 7dfd7f95a5a54851998decd980a3d668 P 745a6616fac446f39b8f5ee7238814c0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 09:59:20.664036 16397 ts_tablet_manager.cc:1905] T 7dfd7f95a5a54851998decd980a3d668 P e785aaa5a6044212b18b8397ce8483e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 09:59:20.664463 16396 raft_consensus.cc:2229] T 7dfd7f95a5a54851998decd980a3d668 P f3295c0ca2c54aea8a83fe6cbfb8bc14 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 09:59:20.666692 16396 raft_consensus.cc:2258] T 7dfd7f95a5a54851998decd980a3d668 P f3295c0ca2c54aea8a83fe6cbfb8bc14 [term 1 FOLLOWER]: Raft consensus is shut down!
Sep 24 09:59:20 dist-test-slave-w8d2 krb5kdc[15852](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243958, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260924 09:59:20.671012 16397 ts_tablet_manager.cc:1918] T 7dfd7f95a5a54851998decd980a3d668 P e785aaa5a6044212b18b8397ce8483e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 09:59:20.671258 16397 log.cc:1196] T 7dfd7f95a5a54851998decd980a3d668 P e785aaa5a6044212b18b8397ce8483e0: Deleting WAL directory at /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/minicluster-data/ts-1/wal/wals/7dfd7f95a5a54851998decd980a3d668
I20260924 09:59:20.671867 16397 ts_tablet_manager.cc:1939] T 7dfd7f95a5a54851998decd980a3d668 P e785aaa5a6044212b18b8397ce8483e0: Deleting consensus metadata
I20260924 09:59:20.671940 16395 ts_tablet_manager.cc:1918] T 7dfd7f95a5a54851998decd980a3d668 P 745a6616fac446f39b8f5ee7238814c0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 09:59:20.672186 16395 log.cc:1196] T 7dfd7f95a5a54851998decd980a3d668 P 745a6616fac446f39b8f5ee7238814c0: Deleting WAL directory at /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/minicluster-data/ts-0/wal/wals/7dfd7f95a5a54851998decd980a3d668
I20260924 09:59:20.672466 16396 ts_tablet_manager.cc:1905] T 7dfd7f95a5a54851998decd980a3d668 P f3295c0ca2c54aea8a83fe6cbfb8bc14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 09:59:20.672824 16395 ts_tablet_manager.cc:1939] T 7dfd7f95a5a54851998decd980a3d668 P 745a6616fac446f39b8f5ee7238814c0: Deleting consensus metadata
I20260924 09:59:20.673676 15884 catalog_manager.cc:4841] TS e785aaa5a6044212b18b8397ce8483e0 (127.0.97.2:40543): tablet 7dfd7f95a5a54851998decd980a3d668 (table test-table [id=44537e0409154114afc862de6c6276cc]) successfully deleted
I20260924 09:59:20.674394 15882 catalog_manager.cc:4841] TS 745a6616fac446f39b8f5ee7238814c0 (127.0.97.1:45337): tablet 7dfd7f95a5a54851998decd980a3d668 (table test-table [id=44537e0409154114afc862de6c6276cc]) successfully deleted
I20260924 09:59:20.677896 16396 ts_tablet_manager.cc:1918] T 7dfd7f95a5a54851998decd980a3d668 P f3295c0ca2c54aea8a83fe6cbfb8bc14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 09:59:20.678027 16396 log.cc:1196] T 7dfd7f95a5a54851998decd980a3d668 P f3295c0ca2c54aea8a83fe6cbfb8bc14: Deleting WAL directory at /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/minicluster-data/ts-2/wal/wals/7dfd7f95a5a54851998decd980a3d668
I20260924 09:59:20.678402 16396 ts_tablet_manager.cc:1939] T 7dfd7f95a5a54851998decd980a3d668 P f3295c0ca2c54aea8a83fe6cbfb8bc14: Deleting consensus metadata
I20260924 09:59:20.679706 15882 catalog_manager.cc:4841] TS f3295c0ca2c54aea8a83fe6cbfb8bc14 (127.0.97.3:39627): tablet 7dfd7f95a5a54851998decd980a3d668 (table test-table [id=44537e0409154114afc862de6c6276cc]) successfully deleted
I20260924 09:59:20.682232 16366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:59:20.646541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35174 (local address 127.0.97.1:45337)
0924 09:59:20.646711 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0924 09:59:20.646718 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 09:59:20.658366 (+ 11648us) server_negotiation.cc:407] Connection header received
0924 09:59:20.658638 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:59:20.658644 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:59:20.658723 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:59:20.658792 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:59:20.660820 (+ 2028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:59:20.662049 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:59:20.664901 (+ 2852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:59:20.665585 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:59:20.665701 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:59:20.675227 (+ 9526us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:59:20.675262 (+ 35us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:59:20.675279 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:59:20.675332 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:59:20.679263 (+ 3931us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:59:20.680132 (+ 869us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:59:20.680142 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:59:20.680146 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:59:20.680355 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:59:20.680705 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:59:20.680710 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:59:20.680713 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:59:20.681131 (+ 418us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:59:20.681408 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0924 09:59:20.681603 (+ 195us) server_negotiation.cc:299] Negotiation successful
0924 09:59:20.681849 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 09:59:20.682885 16081 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 09:59:20.782447 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu with pid 15958
I20260924 09:59:20.797297 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu pid 16092
Sep 24 09:59:20 dist-test-slave-w8d2 krb5kdc[15852](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243958, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260924 09:59:20.810603 16368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:59:20.798065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48570 (local address 127.0.97.2:40543)
0924 09:59:20.798224 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0924 09:59:20.798228 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 09:59:20.798245 (+ 17us) server_negotiation.cc:407] Connection header received
0924 09:59:20.798320 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:59:20.798323 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:59:20.798391 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:59:20.798467 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:59:20.799614 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:59:20.800377 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:59:20.801775 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:59:20.802142 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:59:20.802229 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:59:20.805922 (+ 3693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:59:20.805935 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:59:20.805950 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:59:20.805994 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:59:20.808566 (+ 2572us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:59:20.809132 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:59:20.809138 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:59:20.809140 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:59:20.809248 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:59:20.809511 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:59:20.809514 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:59:20.809516 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:59:20.809841 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:59:20.810036 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0924 09:59:20.810228 (+ 192us) server_negotiation.cc:299] Negotiation successful
0924 09:59:20.810369 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 09:59:20.811352 16215 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 09:59:20.897595 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu with pid 16092
I20260924 09:59:20.910954 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu pid 16225
Sep 24 09:59:20 dist-test-slave-w8d2 krb5kdc[15852](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243958, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260924 09:59:20.925143 16370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:59:20.911712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38716 (local address 127.0.97.3:39627)
0924 09:59:20.911886 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0924 09:59:20.911891 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 09:59:20.911909 (+ 18us) server_negotiation.cc:407] Connection header received
0924 09:59:20.911984 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:59:20.911987 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:59:20.912049 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:59:20.912124 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:59:20.913321 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:59:20.914027 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:59:20.915600 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:59:20.916150 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:59:20.916264 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:59:20.920160 (+ 3896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:59:20.920176 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:59:20.920192 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:59:20.920228 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:59:20.922557 (+ 2329us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:59:20.923179 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:59:20.923189 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:59:20.923192 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:59:20.923355 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:59:20.923702 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:59:20.923708 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:59:20.923711 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:59:20.924052 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:59:20.924315 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0924 09:59:20.924608 (+ 293us) server_negotiation.cc:299] Negotiation successful
0924 09:59:20.924810 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 09:59:20.925688 16348 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 09:59:21.012884 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu with pid 16225
I20260924 09:59:21.027452 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu pid 15865
I20260924 09:59:21.028467 15927 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 09:59:21.134346 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu with pid 15865
I20260924 09:59:41.151525 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
/tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35771
--webserver_interface=127.0.97.62
--webserver_port=40071
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:35771
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 09:59:41.460026 16403 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 09:59:41.460304 16403 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 09:59:41.460433 16403 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 09:59:41.467561 16403 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260924 09:59:41.467656 16403 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260924 09:59:41.467700 16403 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260924 09:59:41.467767 16403 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 09:59:41.467833 16403 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 09:59:41.467926 16403 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 09:59:41.467979 16403 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 09:59:41.468055 16403 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 09:59:41.477764 16403 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:35771
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35771
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=40071
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 09:59:41.479038 16403 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 09:59:41.480365 16403 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 09:59:41.500797 16410 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:59:41.500906 16409 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:59:41.500813 16412 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 09:59:41.501720 16403 server_base.cc:1034] running on GCE node
I20260924 09:59:41.502544 16403 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 09:59:41.504110 16403 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 09:59:41.505332 16403 hybrid_clock.cc:648] HybridClock initialized: now 1790243981505268 us; error 45 us; skew 500 ppm
I20260924 09:59:41.505690 16403 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 09:59:41 dist-test-slave-w8d2 krb5kdc[15852](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243981, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 09:59:41.512773 16403 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260924 09:59:41.514338 16403 webserver.cc:460] Webserver started at http://127.0.97.62:40071/ using document root <none> and password file <none>
I20260924 09:59:41.515271 16403 fs_manager.cc:359] Metadata directory not provided
I20260924 09:59:41.515450 16403 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 09:59:41.521327 16403 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.002s
I20260924 09:59:41.524106 16419 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 09:59:41.525532 16403 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260924 09:59:41.525679 16403 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/minicluster-data/master-0/wal
uuid: "6795636462c1442db8265257ea02ed3e"
format_stamp: "Formatted at 2026-09-24 09:59:18 on dist-test-slave-w8d2"
server_key: "ffcf085de30b7e372d06e0619e30efdc"
server_key_iv: "4fbe7fbf78fb40a8cae744590584df5c"
server_key_version: "encryptionkey@0"
I20260924 09:59:41.526682 16403 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 09:59:41.540941 16403 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 09:59:41.545035 16403 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 09:59:41.545293 16403 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 09:59:41.562166 16403 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:35771
I20260924 09:59:41.562182 16470 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:35771 every 8 connection(s)
I20260924 09:59:41.563702 16403 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/minicluster-data/master-0/data/info.pb
I20260924 09:59:41.570461 16471 sys_catalog.cc:263] Verifying existing consensus state
I20260924 09:59:41.572211 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu as pid 16403
I20260924 09:59:41.573223 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
/tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:45337
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:35771
--webserver_port=44043
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 09:59:41.574574 16471 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6795636462c1442db8265257ea02ed3e: Bootstrap starting.
I20260924 09:59:41.595095 16471 log.cc:824] T 00000000000000000000000000000000 P 6795636462c1442db8265257ea02ed3e: Log is configured to *not* fsync() on all Append() calls
I20260924 09:59:41.606703 16471 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6795636462c1442db8265257ea02ed3e: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 09:59:41.607220 16471 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6795636462c1442db8265257ea02ed3e: Bootstrap complete.
I20260924 09:59:41.616114 16471 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6795636462c1442db8265257ea02ed3e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6795636462c1442db8265257ea02ed3e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35771 } }
I20260924 09:59:41.616914 16471 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6795636462c1442db8265257ea02ed3e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6795636462c1442db8265257ea02ed3e, State: Initialized, Role: FOLLOWER
I20260924 09:59:41.617472 16471 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6795636462c1442db8265257ea02ed3e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6795636462c1442db8265257ea02ed3e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35771 } }
I20260924 09:59:41.617678 16471 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6795636462c1442db8265257ea02ed3e [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 09:59:41.617789 16471 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6795636462c1442db8265257ea02ed3e [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 09:59:41.617935 16471 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6795636462c1442db8265257ea02ed3e [term 1 FOLLOWER]: Advancing to term 2
I20260924 09:59:41.621680 16471 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6795636462c1442db8265257ea02ed3e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6795636462c1442db8265257ea02ed3e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35771 } }
I20260924 09:59:41.622161 16471 leader_election.cc:302] T 00000000000000000000000000000000 P 6795636462c1442db8265257ea02ed3e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6795636462c1442db8265257ea02ed3e; no voters:
I20260924 09:59:41.622736 16471 leader_election.cc:288] T 00000000000000000000000000000000 P 6795636462c1442db8265257ea02ed3e [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 09:59:41.622869 16475 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6795636462c1442db8265257ea02ed3e [term 2 FOLLOWER]: Leader election won for term 2
I20260924 09:59:41.623950 16475 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6795636462c1442db8265257ea02ed3e [term 2 LEADER]: Becoming Leader. State: Replica: 6795636462c1442db8265257ea02ed3e, State: Running, Role: LEADER
I20260924 09:59:41.624524 16475 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6795636462c1442db8265257ea02ed3e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6795636462c1442db8265257ea02ed3e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35771 } }
I20260924 09:59:41.624979 16471 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6795636462c1442db8265257ea02ed3e [sys.catalog]: configured and running, proceeding with master startup.
I20260924 09:59:41.626863 16476 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6795636462c1442db8265257ea02ed3e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "6795636462c1442db8265257ea02ed3e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6795636462c1442db8265257ea02ed3e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35771 } } }
I20260924 09:59:41.627257 16476 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6795636462c1442db8265257ea02ed3e [sys.catalog]: This master's current role is: LEADER
I20260924 09:59:41.631075 16477 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6795636462c1442db8265257ea02ed3e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6795636462c1442db8265257ea02ed3e. Latest consensus state: current_term: 2 leader_uuid: "6795636462c1442db8265257ea02ed3e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6795636462c1442db8265257ea02ed3e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35771 } } }
I20260924 09:59:41.631494 16477 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6795636462c1442db8265257ea02ed3e [sys.catalog]: This master's current role is: LEADER
I20260924 09:59:41.633013 16481 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 09:59:41.641158 16481 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 09:59:41.642608 16481 catalog_manager.cc:1259] Loaded cluster ID: cd7c8363bac24ed7aa68f8264f131800
I20260924 09:59:41.642763 16481 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 09:59:41.646714 16481 catalog_manager.cc:1504] Loading token signing keys...
I20260924 09:59:41.648280 16481 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 6795636462c1442db8265257ea02ed3e: Loaded TSK: 0
I20260924 09:59:41.654157 16492 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260924 09:59:41.680967 16481 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6795636462c1442db8265257ea02ed3e: Generated new TSK 1
I20260924 09:59:41.681380 16481 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 09:59:41.904533 16473 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 09:59:41.904812 16473 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 09:59:41.904924 16473 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 09:59:41.912281 16473 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 09:59:41.912395 16473 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 09:59:41.912536 16473 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260924 09:59:41.922973 16473 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:45337
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=44043
--tserver_master_addrs=127.0.97.62:35771
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 09:59:41.924304 16473 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 09:59:41.925719 16473 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 09:59:41.938709 16498 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:59:41.938675 16499 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 09:59:41.939607 16473 server_base.cc:1034] running on GCE node
W20260924 09:59:41.939718 16501 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 09:59:41.940356 16473 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 09:59:41.941334 16473 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 09:59:41.942545 16473 hybrid_clock.cc:648] HybridClock initialized: now 1790243981942502 us; error 36 us; skew 500 ppm
I20260924 09:59:41.942873 16473 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 09:59:41 dist-test-slave-w8d2 krb5kdc[15852](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243981, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 09:59:41.945897 16473 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260924 09:59:41.947444 16473 webserver.cc:460] Webserver started at http://127.0.97.1:44043/ using document root <none> and password file <none>
I20260924 09:59:41.948112 16473 fs_manager.cc:359] Metadata directory not provided
I20260924 09:59:41.948230 16473 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 09:59:41.953537 16473 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260924 09:59:41.956203 16508 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 09:59:41.957557 16473 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 09:59:41.957708 16473 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/minicluster-data/ts-0/wal
uuid: "745a6616fac446f39b8f5ee7238814c0"
format_stamp: "Formatted at 2026-09-24 09:59:19 on dist-test-slave-w8d2"
server_key: "c3fb61bd489884f59220a26435b8283e"
server_key_iv: "d4548006e18d18dadb99a6ed0dd10a78"
server_key_version: "encryptionkey@0"
I20260924 09:59:41.958639 16473 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 09:59:42.040783 16473 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 09:59:42.044852 16473 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 09:59:42.045094 16473 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 09:59:42.046029 16473 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 09:59:42.047307 16473 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 09:59:42.047400 16473 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 09:59:42.047500 16473 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 09:59:42.047570 16473 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 09:59:42.076766 16473 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:45337
I20260924 09:59:42.076833 16620 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:45337 every 8 connection(s)
I20260924 09:59:42.078106 16473 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/minicluster-data/ts-0/data/info.pb
I20260924 09:59:42.078742 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu as pid 16473
I20260924 09:59:42.079382 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
/tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:40543
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:35771
--webserver_port=45649
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 09:59:42 dist-test-slave-w8d2 krb5kdc[15852](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243981, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 09:59:42.100931 16623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:59:42.081021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:40987 (local address 127.0.97.62:35771)
0924 09:59:42.081545 (+ 524us) server_negotiation.cc:206] Beginning negotiation
0924 09:59:42.081556 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 09:59:42.082222 (+ 666us) server_negotiation.cc:407] Connection header received
0924 09:59:42.084230 (+ 2008us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:59:42.084242 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:59:42.084551 (+ 309us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:59:42.084876 (+ 325us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:59:42.086435 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:59:42.088008 (+ 1573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:59:42.089442 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:59:42.090309 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:59:42.090468 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:59:42.093854 (+ 3386us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:59:42.093864 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:59:42.093881 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:59:42.093945 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:59:42.096796 (+ 2851us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:59:42.097469 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:59:42.097476 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:59:42.097481 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:59:42.097636 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:59:42.098019 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:59:42.098023 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:59:42.098025 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:59:42.098433 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:59:42.098624 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0924 09:59:42.099349 (+ 725us) server_negotiation.cc:299] Negotiation successful
0924 09:59:42.099740 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":227,"threads_started":1}
I20260924 09:59:42.100931 16624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:59:42.081338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35771 (local address 127.0.97.1:40987)
0924 09:59:42.082052 (+ 714us) negotiation.cc:107] Waiting for socket to connect
0924 09:59:42.082087 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 09:59:42.083741 (+ 1654us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:59:42.085031 (+ 1290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:59:42.085041 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:59:42.085584 (+ 543us) client_negotiation.cc:189] Negotiated authn=SASL
0924 09:59:42.086223 (+ 639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:59:42.086246 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:59:42.088157 (+ 1911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:59:42.088161 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:59:42.089315 (+ 1154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:59:42.089326 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:59:42.090480 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:59:42.090484 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:59:42.090665 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:59:42.091294 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 09:59:42.091314 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 09:59:42.093614 (+ 2300us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 09:59:42.096920 (+ 3306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 09:59:42.096929 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 09:59:42.096943 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 09:59:42.097349 (+ 406us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 09:59:42.097738 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 09:59:42.097743 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 09:59:42.097747 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 09:59:42.097920 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 09:59:42.098567 (+ 647us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 09:59:42.098575 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 09:59:42.098887 (+ 312us) client_negotiation.cc:769] Sending connection context
0924 09:59:42.099121 (+ 234us) client_negotiation.cc:240] Negotiation successful
0924 09:59:42.099668 (+ 547us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":282,"threads_started":1}
I20260924 09:59:42.103904 16621 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35771
I20260924 09:59:42.104178 16621 heartbeater.cc:464] Registering TS with master...
I20260924 09:59:42.104833 16621 heartbeater.cc:511] Master 127.0.97.62:35771 requested a full tablet report, sending...
I20260924 09:59:42.106817 16436 ts_manager.cc:194] Registered new tserver with Master: 745a6616fac446f39b8f5ee7238814c0 (127.0.97.1:45337)
I20260924 09:59:42.108239 16436 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:40987
W20260924 09:59:42.395434 16625 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 09:59:42.395730 16625 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 09:59:42.395865 16625 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 09:59:42.402984 16625 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 09:59:42.403103 16625 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 09:59:42.403275 16625 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260924 09:59:42.412933 16625 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:40543
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=45649
--tserver_master_addrs=127.0.97.62:35771
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 09:59:42.414213 16625 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 09:59:42.415532 16625 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 09:59:42.427904 16630 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:59:42.427875 16631 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:59:42.427971 16633 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 09:59:42.428928 16625 server_base.cc:1034] running on GCE node
I20260924 09:59:42.429478 16625 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 09:59:42.430387 16625 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 09:59:42.431571 16625 hybrid_clock.cc:648] HybridClock initialized: now 1790243982431527 us; error 45 us; skew 500 ppm
I20260924 09:59:42.431860 16625 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 09:59:42 dist-test-slave-w8d2 krb5kdc[15852](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243982, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 09:59:42.434888 16625 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260924 09:59:42.436241 16625 webserver.cc:460] Webserver started at http://127.0.97.2:45649/ using document root <none> and password file <none>
I20260924 09:59:42.436856 16625 fs_manager.cc:359] Metadata directory not provided
I20260924 09:59:42.436959 16625 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 09:59:42.441977 16625 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260924 09:59:42.444504 16640 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 09:59:42.445768 16625 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 09:59:42.445911 16625 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/minicluster-data/ts-1/wal
uuid: "e785aaa5a6044212b18b8397ce8483e0"
format_stamp: "Formatted at 2026-09-24 09:59:19 on dist-test-slave-w8d2"
server_key: "0fe5933f754a5200bd6a18cbad61fa62"
server_key_iv: "7b4dd5fbe13dbc0c63f092cfab0b95ef"
server_key_version: "encryptionkey@0"
I20260924 09:59:42.446816 16625 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 09:59:42.466466 16625 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 09:59:42.470548 16625 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 09:59:42.470788 16625 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 09:59:42.471740 16625 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 09:59:42.472978 16625 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 09:59:42.473079 16625 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 09:59:42.473196 16625 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 09:59:42.473263 16625 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 09:59:42.502470 16625 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:40543
I20260924 09:59:42.502517 16752 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:40543 every 8 connection(s)
I20260924 09:59:42.503926 16625 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/minicluster-data/ts-1/data/info.pb
I20260924 09:59:42.511353 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu as pid 16625
I20260924 09:59:42.511940 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
/tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:39627
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:35771
--webserver_port=46083
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 09:59:42 dist-test-slave-w8d2 krb5kdc[15852](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243982, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 09:59:42.524472 16623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:59:42.506258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:40551 (local address 127.0.97.62:35771)
0924 09:59:42.506448 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0924 09:59:42.506453 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 09:59:42.507649 (+ 1196us) server_negotiation.cc:407] Connection header received
0924 09:59:42.509661 (+ 2012us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:59:42.509666 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:59:42.509733 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:59:42.509815 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:59:42.511522 (+ 1707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:59:42.512277 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:59:42.514339 (+ 2062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:59:42.514761 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:59:42.514847 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:59:42.518632 (+ 3785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:59:42.518636 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:59:42.518640 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:59:42.518683 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:59:42.521501 (+ 2818us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:59:42.522177 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:59:42.522182 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:59:42.522185 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:59:42.522316 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:59:42.522719 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:59:42.522723 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:59:42.522726 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:59:42.523046 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:59:42.523182 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0924 09:59:42.523948 (+ 766us) server_negotiation.cc:299] Negotiation successful
0924 09:59:42.524171 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 09:59:42.525319 16755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:59:42.506742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35771 (local address 127.0.97.2:40551)
0924 09:59:42.507452 (+ 710us) negotiation.cc:107] Waiting for socket to connect
0924 09:59:42.507491 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 09:59:42.509356 (+ 1865us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:59:42.510060 (+ 704us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:59:42.510068 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:59:42.510599 (+ 531us) client_negotiation.cc:189] Negotiated authn=SASL
0924 09:59:42.511354 (+ 755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:59:42.511380 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:59:42.512428 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:59:42.512433 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:59:42.514196 (+ 1763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:59:42.514209 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:59:42.514973 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:59:42.514979 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:59:42.515229 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:59:42.515901 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 09:59:42.515932 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0924 09:59:42.518426 (+ 2494us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 09:59:42.521633 (+ 3207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 09:59:42.521640 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 09:59:42.521653 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 09:59:42.522067 (+ 414us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 09:59:42.522415 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 09:59:42.522418 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 09:59:42.522420 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 09:59:42.522621 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 09:59:42.523152 (+ 531us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 09:59:42.523158 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 09:59:42.523593 (+ 435us) client_negotiation.cc:769] Sending connection context
0924 09:59:42.523831 (+ 238us) client_negotiation.cc:240] Negotiation successful
0924 09:59:42.524212 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":263,"threads_started":1}
I20260924 09:59:42.526801 16753 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35771
I20260924 09:59:42.527102 16753 heartbeater.cc:464] Registering TS with master...
I20260924 09:59:42.527954 16753 heartbeater.cc:511] Master 127.0.97.62:35771 requested a full tablet report, sending...
I20260924 09:59:42.529409 16436 ts_manager.cc:194] Registered new tserver with Master: e785aaa5a6044212b18b8397ce8483e0 (127.0.97.2:40543)
I20260924 09:59:42.530270 16436 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:40551
W20260924 09:59:42.831782 16756 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 09:59:42.832072 16756 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 09:59:42.832206 16756 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 09:59:42.839469 16756 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 09:59:42.839583 16756 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 09:59:42.839741 16756 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260924 09:59:42.850515 16756 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:39627
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=46083
--tserver_master_addrs=127.0.97.62:35771
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 09:59:42.851846 16756 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 09:59:42.853097 16756 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 09:59:42.865302 16761 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:59:42.865279 16762 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 09:59:42.865602 16764 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 09:59:42.866007 16756 server_base.cc:1034] running on GCE node
I20260924 09:59:42.866879 16756 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 09:59:42.868052 16756 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 09:59:42.869283 16756 hybrid_clock.cc:648] HybridClock initialized: now 1790243982869219 us; error 46 us; skew 500 ppm
I20260924 09:59:42.869649 16756 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 09:59:42 dist-test-slave-w8d2 krb5kdc[15852](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243982, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 09:59:42.873406 16756 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260924 09:59:42.875144 16756 webserver.cc:460] Webserver started at http://127.0.97.3:46083/ using document root <none> and password file <none>
I20260924 09:59:42.876070 16756 fs_manager.cc:359] Metadata directory not provided
I20260924 09:59:42.876219 16756 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 09:59:42.882279 16756 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.006s sys 0.000s
I20260924 09:59:42.884918 16771 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 09:59:42.886204 16756 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 09:59:42.886345 16756 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/minicluster-data/ts-2/wal
uuid: "f3295c0ca2c54aea8a83fe6cbfb8bc14"
format_stamp: "Formatted at 2026-09-24 09:59:20 on dist-test-slave-w8d2"
server_key: "336612d680ec5c24a7155b536bfeebd2"
server_key_iv: "999a87a783ef1800aab3eeb7bb26335f"
server_key_version: "encryptionkey@0"
I20260924 09:59:42.887303 16756 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 09:59:42.912238 16756 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 09:59:42.917493 16756 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 09:59:42.917778 16756 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 09:59:42.918736 16756 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 09:59:42.920111 16756 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 09:59:42.920204 16756 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 09:59:42.920313 16756 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 09:59:42.920385 16756 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 09:59:42.949424 16756 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:39627
I20260924 09:59:42.949477 16883 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:39627 every 8 connection(s)
I20260924 09:59:42.950914 16756 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/minicluster-data/ts-2/data/info.pb
I20260924 09:59:42.957044 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu as pid 16756
Sep 24 09:59:42 dist-test-slave-w8d2 krb5kdc[15852](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790243982, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 09:59:42.971756 16623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:59:42.953365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:51915 (local address 127.0.97.62:35771)
0924 09:59:42.953533 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0924 09:59:42.953537 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 09:59:42.954654 (+ 1117us) server_negotiation.cc:407] Connection header received
0924 09:59:42.956447 (+ 1793us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:59:42.956451 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:59:42.956519 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:59:42.956607 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:59:42.958502 (+ 1895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:59:42.959344 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:59:42.961501 (+ 2157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:59:42.962210 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:59:42.962336 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:59:42.966251 (+ 3915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:59:42.966256 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:59:42.966259 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:59:42.966288 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:59:42.968872 (+ 2584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:59:42.969536 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:59:42.969541 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:59:42.969545 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:59:42.969700 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:59:42.970084 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:59:42.970089 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:59:42.970092 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:59:42.970429 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:59:42.970571 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0924 09:59:42.971165 (+ 594us) server_negotiation.cc:299] Negotiation successful
0924 09:59:42.971395 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 09:59:42.972894 16886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:59:42.953849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35771 (local address 127.0.97.3:51915)
0924 09:59:42.954506 (+ 657us) negotiation.cc:107] Waiting for socket to connect
0924 09:59:42.954543 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 09:59:42.956212 (+ 1669us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:59:42.956864 (+ 652us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:59:42.956877 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:59:42.957577 (+ 700us) client_negotiation.cc:189] Negotiated authn=SASL
0924 09:59:42.958291 (+ 714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:59:42.958320 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:59:42.959497 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:59:42.959503 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:59:42.961361 (+ 1858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:59:42.961374 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:59:42.962411 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:59:42.962418 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:59:42.962651 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:59:42.963329 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 09:59:42.963364 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0924 09:59:42.966060 (+ 2696us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 09:59:42.968984 (+ 2924us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 09:59:42.968992 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 09:59:42.969008 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 09:59:42.969424 (+ 416us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 09:59:42.969808 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 09:59:42.969811 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 09:59:42.969813 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 09:59:42.969979 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 09:59:42.970527 (+ 548us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 09:59:42.970532 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 09:59:42.970826 (+ 294us) client_negotiation.cc:769] Sending connection context
0924 09:59:42.971053 (+ 227us) client_negotiation.cc:240] Negotiation successful
0924 09:59:42.971583 (+ 530us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":253,"threads_started":1}
I20260924 09:59:42.973446 16888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:59:42.958012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32970 (local address 127.0.97.62:35771)
0924 09:59:42.959592 (+ 1580us) server_negotiation.cc:206] Beginning negotiation
0924 09:59:42.959596 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 09:59:42.959609 (+ 13us) server_negotiation.cc:407] Connection header received
0924 09:59:42.959646 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:59:42.959649 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:59:42.959698 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:59:42.959774 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:59:42.961662 (+ 1888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:59:42.962737 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:59:42.964609 (+ 1872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:59:42.965237 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:59:42.965361 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:59:42.968021 (+ 2660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:59:42.968026 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:59:42.968030 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:59:42.968068 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:59:42.970805 (+ 2737us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:59:42.971609 (+ 804us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:59:42.971612 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:59:42.971614 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:59:42.971714 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:59:42.972124 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:59:42.972129 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:59:42.972131 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:59:42.972433 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:59:42.972671 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0924 09:59:42.972910 (+ 239us) server_negotiation.cc:299] Negotiation successful
0924 09:59:42.973105 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1492,"thread_start_us":1377,"threads_started":1}
I20260924 09:59:42.974301 16884 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35771
I20260924 09:59:42.974589 16884 heartbeater.cc:464] Registering TS with master...
I20260924 09:59:42.975310 16884 heartbeater.cc:511] Master 127.0.97.62:35771 requested a full tablet report, sending...
I20260924 09:59:42.976671 16436 ts_manager.cc:194] Registered new tserver with Master: f3295c0ca2c54aea8a83fe6cbfb8bc14 (127.0.97.3:39627)
I20260924 09:59:42.977461 16436 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:51915
I20260924 09:59:42.987947 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 09:59:42.994347 16888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:59:42.988789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32974 (local address 127.0.97.62:35771)
0924 09:59:42.988964 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0924 09:59:42.988968 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 09:59:42.989407 (+ 439us) server_negotiation.cc:407] Connection header received
0924 09:59:42.989599 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:59:42.989602 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:59:42.989761 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:59:42.989878 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 09:59:42.990305 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:59:42.990988 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:59:42.992510 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:59:42.992946 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:59:42.993054 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:59:42.993516 (+ 462us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 09:59:42.993726 (+ 210us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0924 09:59:42.994082 (+ 356us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:32974: token expired
Metrics: {"server-negotiator.queue_time_us":58}
W20260924 09:59:42.994465 16888 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:32974: token expired
W20260924 09:59:42.994904 16356 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.97.62:35771)
I20260924 09:59:43.006320 16888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:59:42.995428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32980 (local address 127.0.97.62:35771)
0924 09:59:42.995575 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0924 09:59:42.995579 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 09:59:42.995593 (+ 14us) server_negotiation.cc:407] Connection header received
0924 09:59:42.995682 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:59:42.995685 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:59:42.995738 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:59:42.995807 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:59:42.997029 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:59:42.997729 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:59:42.998973 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:59:42.999398 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:59:42.999479 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:59:43.001751 (+ 2272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:59:43.001755 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:59:43.001758 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:59:43.001789 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:59:43.004220 (+ 2431us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:59:43.004793 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:59:43.004797 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:59:43.004799 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:59:43.004910 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:59:43.005168 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:59:43.005171 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:59:43.005173 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:59:43.005461 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:59:43.005639 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0924 09:59:43.005877 (+ 238us) server_negotiation.cc:299] Negotiation successful
0924 09:59:43.006030 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 09:59:43.009603 16436 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32980:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 09:59:43.016458 16436 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 09:59:43.049909 16898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:59:43.029247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:40543 (local address 127.0.0.1:33002)
0924 09:59:43.040857 (+ 11610us) negotiation.cc:107] Waiting for socket to connect
0924 09:59:43.040879 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 09:59:43.041025 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:59:43.042304 (+ 1279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:59:43.042313 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:59:43.042347 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 09:59:43.042779 (+ 432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:59:43.042792 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:59:43.044581 (+ 1789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:59:43.044586 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:59:43.047054 (+ 2468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:59:43.047068 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:59:43.048902 (+ 1834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:59:43.048908 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:59:43.049096 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:59:43.049160 (+ 64us) client_negotiation.cc:769] Sending connection context
0924 09:59:43.049294 (+ 134us) client_negotiation.cc:240] Negotiation successful
0924 09:59:43.049466 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11455,"mutex_wait_us":40,"thread_start_us":13604,"threads_started":1}
I20260924 09:59:43.050402 16897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:59:43.029849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33002 (local address 127.0.97.2:40543)
0924 09:59:43.038000 (+ 8151us) server_negotiation.cc:206] Beginning negotiation
0924 09:59:43.038012 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 09:59:43.041180 (+ 3168us) server_negotiation.cc:407] Connection header received
0924 09:59:43.041312 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:59:43.041321 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:59:43.041630 (+ 309us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:59:43.041848 (+ 218us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 09:59:43.043041 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:59:43.044449 (+ 1408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:59:43.047219 (+ 2770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:59:43.048682 (+ 1463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:59:43.048810 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:59:43.049009 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0924 09:59:43.049337 (+ 328us) server_negotiation.cc:299] Negotiation successful
0924 09:59:43.049497 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7930,"thread_start_us":7806,"threads_started":1}
I20260924 09:59:43.050601 16894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:59:43.028521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45337 (local address 127.0.0.1:44556)
0924 09:59:43.029478 (+ 957us) negotiation.cc:107] Waiting for socket to connect
0924 09:59:43.029515 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 09:59:43.029721 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:59:43.044057 (+ 14336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:59:43.044061 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:59:43.044104 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 09:59:43.044411 (+ 307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:59:43.044422 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:59:43.046403 (+ 1981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:59:43.046407 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:59:43.048119 (+ 1712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:59:43.048130 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:59:43.049913 (+ 1783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:59:43.049922 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:59:43.050083 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:59:43.050120 (+ 37us) client_negotiation.cc:769] Sending connection context
0924 09:59:43.050202 (+ 82us) client_negotiation.cc:240] Negotiation successful
0924 09:59:43.050317 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":686,"mutex_wait_us":20,"thread_start_us":527,"threads_started":1}
I20260924 09:59:43.051309 16895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:59:43.028718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44556 (local address 127.0.97.1:45337)
0924 09:59:43.043414 (+ 14696us) server_negotiation.cc:206] Beginning negotiation
0924 09:59:43.043423 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 09:59:43.043452 (+ 29us) server_negotiation.cc:407] Connection header received
0924 09:59:43.043545 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:59:43.043552 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:59:43.043838 (+ 286us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:59:43.044635 (+ 797us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 09:59:43.044862 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:59:43.046266 (+ 1404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:59:43.048265 (+ 1999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:59:43.049406 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:59:43.049516 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:59:43.049701 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0924 09:59:43.050822 (+ 1121us) server_negotiation.cc:299] Negotiation successful
0924 09:59:43.051022 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14538,"thread_start_us":14526,"threads_started":1}
I20260924 09:59:43.051800 16896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:59:43.029966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:39627 (local address 127.0.0.1:55656)
0924 09:59:43.043115 (+ 13149us) negotiation.cc:107] Waiting for socket to connect
0924 09:59:43.043133 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 09:59:43.043475 (+ 342us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:59:43.044651 (+ 1176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:59:43.044655 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:59:43.044679 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 09:59:43.045073 (+ 394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:59:43.045084 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:59:43.046766 (+ 1682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:59:43.046770 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:59:43.048884 (+ 2114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:59:43.048897 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:59:43.051024 (+ 2127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:59:43.051030 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:59:43.051181 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:59:43.051210 (+ 29us) client_negotiation.cc:769] Sending connection context
0924 09:59:43.051308 (+ 98us) client_negotiation.cc:240] Negotiation successful
0924 09:59:43.051422 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13032,"mutex_wait_us":1,"thread_start_us":12076,"threads_started":1}
I20260924 09:59:43.052424 16899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:59:43.030294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55656 (local address 127.0.97.3:39627)
0924 09:59:43.044082 (+ 13788us) server_negotiation.cc:206] Beginning negotiation
0924 09:59:43.044090 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 09:59:43.044126 (+ 36us) server_negotiation.cc:407] Connection header received
0924 09:59:43.044235 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:59:43.044243 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:59:43.044471 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:59:43.044645 (+ 174us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 09:59:43.045260 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:59:43.046642 (+ 1382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:59:43.049049 (+ 2407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:59:43.050566 (+ 1517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:59:43.050679 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:59:43.050896 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0924 09:59:43.051993 (+ 1097us) server_negotiation.cc:299] Negotiation successful
0924 09:59:43.052142 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13621,"thread_start_us":13482,"threads_started":1}
I20260924 09:59:43.054590 16688 tablet_service.cc:1467] Processing CreateTablet for tablet 0443f524c3334c78953dbeb19f2a700a (DEFAULT_TABLE table=test-table [id=7d793cbdea4e49d28c9af09f1eb94edb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 09:59:43.054778 16556 tablet_service.cc:1467] Processing CreateTablet for tablet 0443f524c3334c78953dbeb19f2a700a (DEFAULT_TABLE table=test-table [id=7d793cbdea4e49d28c9af09f1eb94edb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 09:59:43.054911 16819 tablet_service.cc:1467] Processing CreateTablet for tablet 0443f524c3334c78953dbeb19f2a700a (DEFAULT_TABLE table=test-table [id=7d793cbdea4e49d28c9af09f1eb94edb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 09:59:43.055997 16819 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0443f524c3334c78953dbeb19f2a700a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 09:59:43.056011 16556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0443f524c3334c78953dbeb19f2a700a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 09:59:43.056011 16688 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0443f524c3334c78953dbeb19f2a700a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 09:59:43.068967 16900 tablet_bootstrap.cc:492] T 0443f524c3334c78953dbeb19f2a700a P e785aaa5a6044212b18b8397ce8483e0: Bootstrap starting.
I20260924 09:59:43.071555 16902 tablet_bootstrap.cc:492] T 0443f524c3334c78953dbeb19f2a700a P 745a6616fac446f39b8f5ee7238814c0: Bootstrap starting.
I20260924 09:59:43.072453 16901 tablet_bootstrap.cc:492] T 0443f524c3334c78953dbeb19f2a700a P f3295c0ca2c54aea8a83fe6cbfb8bc14: Bootstrap starting.
I20260924 09:59:43.072619 16900 tablet_bootstrap.cc:654] T 0443f524c3334c78953dbeb19f2a700a P e785aaa5a6044212b18b8397ce8483e0: Neither blocks nor log segments found. Creating new log.
I20260924 09:59:43.073832 16900 log.cc:824] T 0443f524c3334c78953dbeb19f2a700a P e785aaa5a6044212b18b8397ce8483e0: Log is configured to *not* fsync() on all Append() calls
I20260924 09:59:43.074743 16902 tablet_bootstrap.cc:654] T 0443f524c3334c78953dbeb19f2a700a P 745a6616fac446f39b8f5ee7238814c0: Neither blocks nor log segments found. Creating new log.
I20260924 09:59:43.075276 16901 tablet_bootstrap.cc:654] T 0443f524c3334c78953dbeb19f2a700a P f3295c0ca2c54aea8a83fe6cbfb8bc14: Neither blocks nor log segments found. Creating new log.
I20260924 09:59:43.075898 16902 log.cc:824] T 0443f524c3334c78953dbeb19f2a700a P 745a6616fac446f39b8f5ee7238814c0: Log is configured to *not* fsync() on all Append() calls
I20260924 09:59:43.076176 16901 log.cc:824] T 0443f524c3334c78953dbeb19f2a700a P f3295c0ca2c54aea8a83fe6cbfb8bc14: Log is configured to *not* fsync() on all Append() calls
I20260924 09:59:43.076844 16900 tablet_bootstrap.cc:492] T 0443f524c3334c78953dbeb19f2a700a P e785aaa5a6044212b18b8397ce8483e0: No bootstrap required, opened a new log
I20260924 09:59:43.077122 16900 ts_tablet_manager.cc:1397] T 0443f524c3334c78953dbeb19f2a700a P e785aaa5a6044212b18b8397ce8483e0: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260924 09:59:43.079182 16901 tablet_bootstrap.cc:492] T 0443f524c3334c78953dbeb19f2a700a P f3295c0ca2c54aea8a83fe6cbfb8bc14: No bootstrap required, opened a new log
I20260924 09:59:43.079576 16901 ts_tablet_manager.cc:1397] T 0443f524c3334c78953dbeb19f2a700a P f3295c0ca2c54aea8a83fe6cbfb8bc14: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260924 09:59:43.088634 16900 raft_consensus.cc:348] T 0443f524c3334c78953dbeb19f2a700a P e785aaa5a6044212b18b8397ce8483e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745a6616fac446f39b8f5ee7238814c0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45337 } } peers { permanent_uuid: "f3295c0ca2c54aea8a83fe6cbfb8bc14" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39627 } } peers { permanent_uuid: "e785aaa5a6044212b18b8397ce8483e0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40543 } }
I20260924 09:59:43.089098 16900 raft_consensus.cc:374] T 0443f524c3334c78953dbeb19f2a700a P e785aaa5a6044212b18b8397ce8483e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 09:59:43.089213 16900 raft_consensus.cc:729] T 0443f524c3334c78953dbeb19f2a700a P e785aaa5a6044212b18b8397ce8483e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e785aaa5a6044212b18b8397ce8483e0, State: Initialized, Role: FOLLOWER
I20260924 09:59:43.090456 16900 consensus_queue.cc:260] T 0443f524c3334c78953dbeb19f2a700a P e785aaa5a6044212b18b8397ce8483e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745a6616fac446f39b8f5ee7238814c0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45337 } } peers { permanent_uuid: "f3295c0ca2c54aea8a83fe6cbfb8bc14" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39627 } } peers { permanent_uuid: "e785aaa5a6044212b18b8397ce8483e0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40543 } }
I20260924 09:59:43.090507 16901 raft_consensus.cc:348] T 0443f524c3334c78953dbeb19f2a700a P f3295c0ca2c54aea8a83fe6cbfb8bc14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745a6616fac446f39b8f5ee7238814c0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45337 } } peers { permanent_uuid: "f3295c0ca2c54aea8a83fe6cbfb8bc14" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39627 } } peers { permanent_uuid: "e785aaa5a6044212b18b8397ce8483e0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40543 } }
I20260924 09:59:43.091128 16901 raft_consensus.cc:374] T 0443f524c3334c78953dbeb19f2a700a P f3295c0ca2c54aea8a83fe6cbfb8bc14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 09:59:43.091291 16901 raft_consensus.cc:729] T 0443f524c3334c78953dbeb19f2a700a P f3295c0ca2c54aea8a83fe6cbfb8bc14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3295c0ca2c54aea8a83fe6cbfb8bc14, State: Initialized, Role: FOLLOWER
I20260924 09:59:43.091917 16901 consensus_queue.cc:260] T 0443f524c3334c78953dbeb19f2a700a P f3295c0ca2c54aea8a83fe6cbfb8bc14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745a6616fac446f39b8f5ee7238814c0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45337 } } peers { permanent_uuid: "f3295c0ca2c54aea8a83fe6cbfb8bc14" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39627 } } peers { permanent_uuid: "e785aaa5a6044212b18b8397ce8483e0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40543 } }
I20260924 09:59:43.092604 16902 tablet_bootstrap.cc:492] T 0443f524c3334c78953dbeb19f2a700a P 745a6616fac446f39b8f5ee7238814c0: No bootstrap required, opened a new log
I20260924 09:59:43.092895 16902 ts_tablet_manager.cc:1397] T 0443f524c3334c78953dbeb19f2a700a P 745a6616fac446f39b8f5ee7238814c0: Time spent bootstrapping tablet: real 0.022s user 0.012s sys 0.007s
I20260924 09:59:43.093099 16884 heartbeater.cc:503] Master 127.0.97.62:35771 was elected leader, sending a full tablet report...
I20260924 09:59:43.093827 16901 ts_tablet_manager.cc:1428] T 0443f524c3334c78953dbeb19f2a700a P f3295c0ca2c54aea8a83fe6cbfb8bc14: Time spent starting tablet: real 0.014s user 0.009s sys 0.004s
I20260924 09:59:43.095898 16753 heartbeater.cc:503] Master 127.0.97.62:35771 was elected leader, sending a full tablet report...
I20260924 09:59:43.096515 16900 ts_tablet_manager.cc:1428] T 0443f524c3334c78953dbeb19f2a700a P e785aaa5a6044212b18b8397ce8483e0: Time spent starting tablet: real 0.019s user 0.013s sys 0.003s
I20260924 09:59:43.101282 16902 raft_consensus.cc:348] T 0443f524c3334c78953dbeb19f2a700a P 745a6616fac446f39b8f5ee7238814c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745a6616fac446f39b8f5ee7238814c0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45337 } } peers { permanent_uuid: "f3295c0ca2c54aea8a83fe6cbfb8bc14" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39627 } } peers { permanent_uuid: "e785aaa5a6044212b18b8397ce8483e0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40543 } }
I20260924 09:59:43.101608 16902 raft_consensus.cc:374] T 0443f524c3334c78953dbeb19f2a700a P 745a6616fac446f39b8f5ee7238814c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 09:59:43.101706 16902 raft_consensus.cc:729] T 0443f524c3334c78953dbeb19f2a700a P 745a6616fac446f39b8f5ee7238814c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 745a6616fac446f39b8f5ee7238814c0, State: Initialized, Role: FOLLOWER
I20260924 09:59:43.102210 16902 consensus_queue.cc:260] T 0443f524c3334c78953dbeb19f2a700a P 745a6616fac446f39b8f5ee7238814c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745a6616fac446f39b8f5ee7238814c0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45337 } } peers { permanent_uuid: "f3295c0ca2c54aea8a83fe6cbfb8bc14" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39627 } } peers { permanent_uuid: "e785aaa5a6044212b18b8397ce8483e0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40543 } }
I20260924 09:59:43.103127 16621 heartbeater.cc:503] Master 127.0.97.62:35771 was elected leader, sending a full tablet report...
I20260924 09:59:43.103721 16902 ts_tablet_manager.cc:1428] T 0443f524c3334c78953dbeb19f2a700a P 745a6616fac446f39b8f5ee7238814c0: Time spent starting tablet: real 0.011s user 0.011s sys 0.000s
W20260924 09:59:43.202436 16885 tablet.cc:2369] T 0443f524c3334c78953dbeb19f2a700a P f3295c0ca2c54aea8a83fe6cbfb8bc14: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 09:59:43.255725 16754 tablet.cc:2369] T 0443f524c3334c78953dbeb19f2a700a P e785aaa5a6044212b18b8397ce8483e0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 09:59:43.330475 16622 tablet.cc:2369] T 0443f524c3334c78953dbeb19f2a700a P 745a6616fac446f39b8f5ee7238814c0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 09:59:43.365808 16906 raft_consensus.cc:482] T 0443f524c3334c78953dbeb19f2a700a P e785aaa5a6044212b18b8397ce8483e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 09:59:43.366115 16906 raft_consensus.cc:504] T 0443f524c3334c78953dbeb19f2a700a P e785aaa5a6044212b18b8397ce8483e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745a6616fac446f39b8f5ee7238814c0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45337 } } peers { permanent_uuid: "f3295c0ca2c54aea8a83fe6cbfb8bc14" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39627 } } peers { permanent_uuid: "e785aaa5a6044212b18b8397ce8483e0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40543 } }
I20260924 09:59:43.367949 16906 leader_election.cc:288] T 0443f524c3334c78953dbeb19f2a700a P e785aaa5a6044212b18b8397ce8483e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 745a6616fac446f39b8f5ee7238814c0 (127.0.97.1:45337), f3295c0ca2c54aea8a83fe6cbfb8bc14 (127.0.97.3:39627)
I20260924 09:59:43.374150 16895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:59:43.368467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:48623 (local address 127.0.97.1:45337)
0924 09:59:43.368691 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0924 09:59:43.368698 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 09:59:43.369110 (+ 412us) server_negotiation.cc:407] Connection header received
0924 09:59:43.369356 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:59:43.369363 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:59:43.369448 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:59:43.369541 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 09:59:43.370084 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:59:43.370785 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:59:43.372517 (+ 1732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:59:43.373297 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:59:43.373412 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:59:43.373506 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0924 09:59:43.373774 (+ 268us) server_negotiation.cc:299] Negotiation successful
0924 09:59:43.373896 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260924 09:59:43.374230 16909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:59:43.368309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45337 (local address 127.0.97.2:48623)
0924 09:59:43.369001 (+ 692us) negotiation.cc:107] Waiting for socket to connect
0924 09:59:43.369016 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 09:59:43.369164 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:59:43.369600 (+ 436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:59:43.369604 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:59:43.369645 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 09:59:43.369948 (+ 303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:59:43.369957 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:59:43.370930 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:59:43.370934 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:59:43.372365 (+ 1431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:59:43.372377 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:59:43.373500 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:59:43.373506 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:59:43.373637 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:59:43.373661 (+ 24us) client_negotiation.cc:769] Sending connection context
0924 09:59:43.373739 (+ 78us) client_negotiation.cc:240] Negotiation successful
0924 09:59:43.373872 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":578,"thread_start_us":395,"threads_started":1}
I20260924 09:59:43.374497 16899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:59:43.369374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:54377 (local address 127.0.97.3:39627)
0924 09:59:43.369564 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0924 09:59:43.369569 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 09:59:43.369790 (+ 221us) server_negotiation.cc:407] Connection header received
0924 09:59:43.369944 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:59:43.369948 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:59:43.370004 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:59:43.370074 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 09:59:43.370594 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:59:43.371308 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:59:43.372901 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:59:43.373652 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:59:43.373764 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:59:43.373859 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0924 09:59:43.374144 (+ 285us) server_negotiation.cc:299] Negotiation successful
0924 09:59:43.374262 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260924 09:59:43.374543 16910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:59:43.369103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:39627 (local address 127.0.97.2:54377)
0924 09:59:43.369678 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0924 09:59:43.369692 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 09:59:43.369817 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 09:59:43.370110 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 09:59:43.370114 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 09:59:43.370138 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 09:59:43.370453 (+ 315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:59:43.370462 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:59:43.371422 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:59:43.371427 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:59:43.372760 (+ 1333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 09:59:43.372771 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 09:59:43.373853 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 09:59:43.373859 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 09:59:43.374008 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:59:43.374030 (+ 22us) client_negotiation.cc:769] Sending connection context
0924 09:59:43.374104 (+ 74us) client_negotiation.cc:240] Negotiation successful
0924 09:59:43.374228 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":298,"threads_started":1}
I20260924 09:59:43.375192 16576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0443f524c3334c78953dbeb19f2a700a" candidate_uuid: "e785aaa5a6044212b18b8397ce8483e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "745a6616fac446f39b8f5ee7238814c0" is_pre_election: true
I20260924 09:59:43.375394 16839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0443f524c3334c78953dbeb19f2a700a" candidate_uuid: "e785aaa5a6044212b18b8397ce8483e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3295c0ca2c54aea8a83fe6cbfb8bc14" is_pre_election: true
I20260924 09:59:43.375738 16576 raft_consensus.cc:2454] T 0443f524c3334c78953dbeb19f2a700a P 745a6616fac446f39b8f5ee7238814c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e785aaa5a6044212b18b8397ce8483e0 in term 0.
I20260924 09:59:43.375824 16839 raft_consensus.cc:2454] T 0443f524c3334c78953dbeb19f2a700a P f3295c0ca2c54aea8a83fe6cbfb8bc14 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e785aaa5a6044212b18b8397ce8483e0 in term 0.
I20260924 09:59:43.376389 16642 leader_election.cc:302] T 0443f524c3334c78953dbeb19f2a700a P e785aaa5a6044212b18b8397ce8483e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 745a6616fac446f39b8f5ee7238814c0, e785aaa5a6044212b18b8397ce8483e0; no voters:
I20260924 09:59:43.376730 16906 raft_consensus.cc:2781] T 0443f524c3334c78953dbeb19f2a700a P e785aaa5a6044212b18b8397ce8483e0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 09:59:43.376878 16906 raft_consensus.cc:482] T 0443f524c3334c78953dbeb19f2a700a P e785aaa5a6044212b18b8397ce8483e0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 09:59:43.376979 16906 raft_consensus.cc:3037] T 0443f524c3334c78953dbeb19f2a700a P e785aaa5a6044212b18b8397ce8483e0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 09:59:43.379042 16906 raft_consensus.cc:504] T 0443f524c3334c78953dbeb19f2a700a P e785aaa5a6044212b18b8397ce8483e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745a6616fac446f39b8f5ee7238814c0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45337 } } peers { permanent_uuid: "f3295c0ca2c54aea8a83fe6cbfb8bc14" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39627 } } peers { permanent_uuid: "e785aaa5a6044212b18b8397ce8483e0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40543 } }
I20260924 09:59:43.379683 16906 leader_election.cc:288] T 0443f524c3334c78953dbeb19f2a700a P e785aaa5a6044212b18b8397ce8483e0 [CANDIDATE]: Term 1 election: Requested vote from peers 745a6616fac446f39b8f5ee7238814c0 (127.0.97.1:45337), f3295c0ca2c54aea8a83fe6cbfb8bc14 (127.0.97.3:39627)
I20260924 09:59:43.380102 16576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0443f524c3334c78953dbeb19f2a700a" candidate_uuid: "e785aaa5a6044212b18b8397ce8483e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "745a6616fac446f39b8f5ee7238814c0"
I20260924 09:59:43.380167 16839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0443f524c3334c78953dbeb19f2a700a" candidate_uuid: "e785aaa5a6044212b18b8397ce8483e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3295c0ca2c54aea8a83fe6cbfb8bc14"
I20260924 09:59:43.380283 16576 raft_consensus.cc:3037] T 0443f524c3334c78953dbeb19f2a700a P 745a6616fac446f39b8f5ee7238814c0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 09:59:43.380365 16839 raft_consensus.cc:3037] T 0443f524c3334c78953dbeb19f2a700a P f3295c0ca2c54aea8a83fe6cbfb8bc14 [term 0 FOLLOWER]: Advancing to term 1
I20260924 09:59:43.382381 16576 raft_consensus.cc:2454] T 0443f524c3334c78953dbeb19f2a700a P 745a6616fac446f39b8f5ee7238814c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e785aaa5a6044212b18b8397ce8483e0 in term 1.
I20260924 09:59:43.382391 16839 raft_consensus.cc:2454] T 0443f524c3334c78953dbeb19f2a700a P f3295c0ca2c54aea8a83fe6cbfb8bc14 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e785aaa5a6044212b18b8397ce8483e0 in term 1.
I20260924 09:59:43.382716 16642 leader_election.cc:302] T 0443f524c3334c78953dbeb19f2a700a P e785aaa5a6044212b18b8397ce8483e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e785aaa5a6044212b18b8397ce8483e0, f3295c0ca2c54aea8a83fe6cbfb8bc14; no voters:
I20260924 09:59:43.382963 16906 raft_consensus.cc:2781] T 0443f524c3334c78953dbeb19f2a700a P e785aaa5a6044212b18b8397ce8483e0 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 09:59:43.383402 16906 raft_consensus.cc:686] T 0443f524c3334c78953dbeb19f2a700a P e785aaa5a6044212b18b8397ce8483e0 [term 1 LEADER]: Becoming Leader. State: Replica: e785aaa5a6044212b18b8397ce8483e0, State: Running, Role: LEADER
I20260924 09:59:43.383859 16906 consensus_queue.cc:237] T 0443f524c3334c78953dbeb19f2a700a P e785aaa5a6044212b18b8397ce8483e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745a6616fac446f39b8f5ee7238814c0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45337 } } peers { permanent_uuid: "f3295c0ca2c54aea8a83fe6cbfb8bc14" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39627 } } peers { permanent_uuid: "e785aaa5a6044212b18b8397ce8483e0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40543 } }
I20260924 09:59:43.388105 16435 catalog_manager.cc:5495] T 0443f524c3334c78953dbeb19f2a700a P e785aaa5a6044212b18b8397ce8483e0 reported cstate change: term changed from 0 to 1, leader changed from <none> to e785aaa5a6044212b18b8397ce8483e0 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "e785aaa5a6044212b18b8397ce8483e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745a6616fac446f39b8f5ee7238814c0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45337 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f3295c0ca2c54aea8a83fe6cbfb8bc14" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39627 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e785aaa5a6044212b18b8397ce8483e0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40543 } health_report { overall_health: HEALTHY } } }
I20260924 09:59:43.403366 16897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:59:43.398198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33010 (local address 127.0.97.2:40543)
0924 09:59:43.398377 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0924 09:59:43.398383 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 09:59:43.398464 (+ 81us) server_negotiation.cc:407] Connection header received
0924 09:59:43.398669 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:59:43.398673 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:59:43.398747 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:59:43.398870 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 09:59:43.399202 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:59:43.399950 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:59:43.401295 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:59:43.401678 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:59:43.401783 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:59:43.402273 (+ 490us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 09:59:43.402750 (+ 477us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 09:59:43.402885 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0924 09:59:43.402984 (+ 99us) server_negotiation.cc:299] Negotiation successful
0924 09:59:43.403062 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 09:59:43.410946 16839 raft_consensus.cc:1261] T 0443f524c3334c78953dbeb19f2a700a P f3295c0ca2c54aea8a83fe6cbfb8bc14 [term 1 FOLLOWER]: Refusing update from remote peer e785aaa5a6044212b18b8397ce8483e0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 09:59:43.410961 16576 raft_consensus.cc:1261] T 0443f524c3334c78953dbeb19f2a700a P 745a6616fac446f39b8f5ee7238814c0 [term 1 FOLLOWER]: Refusing update from remote peer e785aaa5a6044212b18b8397ce8483e0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 09:59:43.411846 16911 consensus_queue.cc:1035] T 0443f524c3334c78953dbeb19f2a700a P e785aaa5a6044212b18b8397ce8483e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f3295c0ca2c54aea8a83fe6cbfb8bc14" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39627 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 09:59:43.412164 16906 consensus_queue.cc:1035] T 0443f524c3334c78953dbeb19f2a700a P e785aaa5a6044212b18b8397ce8483e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "745a6616fac446f39b8f5ee7238814c0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 09:59:43.434068 16435 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32980:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 09:59:43.434348 16435 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32980:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 09:59:43.437779 16435 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6795636462c1442db8265257ea02ed3e: Sending DeleteTablet for 3 replicas of tablet 0443f524c3334c78953dbeb19f2a700a
I20260924 09:59:43.438920 16556 tablet_service.cc:1514] Processing DeleteTablet for tablet 0443f524c3334c78953dbeb19f2a700a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 09:59:43 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:44556
I20260924 09:59:43.439006 16688 tablet_service.cc:1514] Processing DeleteTablet for tablet 0443f524c3334c78953dbeb19f2a700a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 09:59:43 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:33002
I20260924 09:59:43.439658 16819 tablet_service.cc:1514] Processing DeleteTablet for tablet 0443f524c3334c78953dbeb19f2a700a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 09:59:43 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:55656
I20260924 09:59:43.440003 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu pid 16473
I20260924 09:59:43.450057 16925 tablet_replica.cc:330] T 0443f524c3334c78953dbeb19f2a700a P f3295c0ca2c54aea8a83fe6cbfb8bc14: stopping tablet replica
I20260924 09:59:43.451306 16925 raft_consensus.cc:2229] T 0443f524c3334c78953dbeb19f2a700a P f3295c0ca2c54aea8a83fe6cbfb8bc14 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 09:59:43.452386 16925 raft_consensus.cc:2258] T 0443f524c3334c78953dbeb19f2a700a P f3295c0ca2c54aea8a83fe6cbfb8bc14 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 09:59:43.456838 16923 tablet_replica.cc:330] T 0443f524c3334c78953dbeb19f2a700a P e785aaa5a6044212b18b8397ce8483e0: stopping tablet replica
I20260924 09:59:43.457193 16923 raft_consensus.cc:2229] T 0443f524c3334c78953dbeb19f2a700a P e785aaa5a6044212b18b8397ce8483e0 [term 1 LEADER]: Raft consensus shutting down.
I20260924 09:59:43.457374 16895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:59:43.440864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44558 (local address 127.0.97.1:45337)
0924 09:59:43.441025 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0924 09:59:43.441030 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 09:59:43.441049 (+ 19us) server_negotiation.cc:407] Connection header received
0924 09:59:43.441199 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:59:43.441205 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:59:43.441286 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:59:43.441374 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:59:43.443160 (+ 1786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:59:43.443964 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:59:43.445807 (+ 1843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:59:43.446227 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:59:43.446335 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:59:43.449639 (+ 3304us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:59:43.449659 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:59:43.449679 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:59:43.449723 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:59:43.453795 (+ 4072us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:59:43.454685 (+ 890us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:59:43.454694 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:59:43.454698 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:59:43.454861 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:59:43.455346 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:59:43.455351 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:59:43.455355 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:59:43.455915 (+ 560us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:59:43.456184 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0924 09:59:43.456771 (+ 587us) server_negotiation.cc:299] Negotiation successful
0924 09:59:43.456989 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 09:59:43.457779 16923 raft_consensus.cc:2258] T 0443f524c3334c78953dbeb19f2a700a P e785aaa5a6044212b18b8397ce8483e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 09:59:43.458670 16925 ts_tablet_manager.cc:1905] T 0443f524c3334c78953dbeb19f2a700a P f3295c0ca2c54aea8a83fe6cbfb8bc14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 09:59:43.459010 16616 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 09:59:43.463925 16923 ts_tablet_manager.cc:1905] T 0443f524c3334c78953dbeb19f2a700a P e785aaa5a6044212b18b8397ce8483e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 09:59:43.467180 16925 ts_tablet_manager.cc:1918] T 0443f524c3334c78953dbeb19f2a700a P f3295c0ca2c54aea8a83fe6cbfb8bc14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 09:59:43.467468 16925 log.cc:1196] T 0443f524c3334c78953dbeb19f2a700a P f3295c0ca2c54aea8a83fe6cbfb8bc14: Deleting WAL directory at /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/minicluster-data/ts-2/wal/wals/0443f524c3334c78953dbeb19f2a700a
I20260924 09:59:43.468051 16925 ts_tablet_manager.cc:1939] T 0443f524c3334c78953dbeb19f2a700a P f3295c0ca2c54aea8a83fe6cbfb8bc14: Deleting consensus metadata
I20260924 09:59:43.470381 16421 catalog_manager.cc:4841] TS f3295c0ca2c54aea8a83fe6cbfb8bc14 (127.0.97.3:39627): tablet 0443f524c3334c78953dbeb19f2a700a (table test-table [id=7d793cbdea4e49d28c9af09f1eb94edb]) successfully deleted
I20260924 09:59:43.473100 16923 ts_tablet_manager.cc:1918] T 0443f524c3334c78953dbeb19f2a700a P e785aaa5a6044212b18b8397ce8483e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 09:59:43.473342 16923 log.cc:1196] T 0443f524c3334c78953dbeb19f2a700a P e785aaa5a6044212b18b8397ce8483e0: Deleting WAL directory at /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/minicluster-data/ts-1/wal/wals/0443f524c3334c78953dbeb19f2a700a
I20260924 09:59:43.474206 16923 ts_tablet_manager.cc:1939] T 0443f524c3334c78953dbeb19f2a700a P e785aaa5a6044212b18b8397ce8483e0: Deleting consensus metadata
I20260924 09:59:43.475775 16423 catalog_manager.cc:4841] TS e785aaa5a6044212b18b8397ce8483e0 (127.0.97.2:40543): tablet 0443f524c3334c78953dbeb19f2a700a (table test-table [id=7d793cbdea4e49d28c9af09f1eb94edb]) successfully deleted
I20260924 09:59:43.617353 16924 tablet_replica.cc:330] T 0443f524c3334c78953dbeb19f2a700a P 745a6616fac446f39b8f5ee7238814c0: stopping tablet replica
I20260924 09:59:43.617448 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu with pid 16473
W20260924 09:59:43.631718 16421 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:45337 (error 108)
I20260924 09:59:43.631906 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu pid 16625
I20260924 09:59:43.633788 16928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:59:43.633075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45337 (local address 127.0.0.1:44574)
0924 09:59:43.633484 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0924 09:59:43.633630 (+ 146us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45337: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":220,"threads_started":1}
W20260924 09:59:43.634096 16421 catalog_manager.cc:4584] TS 745a6616fac446f39b8f5ee7238814c0 (127.0.97.1:45337): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0443f524c3334c78953dbeb19f2a700a: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45337: connect: Connection refused (error 111)
I20260924 09:59:43.647073 16897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:59:43.632591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33012 (local address 127.0.97.2:40543)
0924 09:59:43.632793 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0924 09:59:43.632798 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 09:59:43.633030 (+ 232us) server_negotiation.cc:407] Connection header received
0924 09:59:43.633211 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:59:43.633217 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:59:43.633327 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:59:43.633424 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:59:43.635189 (+ 1765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:59:43.636007 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:59:43.637872 (+ 1865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:59:43.638284 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:59:43.638372 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:59:43.641515 (+ 3143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:59:43.641527 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:59:43.641541 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:59:43.641575 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:59:43.644437 (+ 2862us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:59:43.645219 (+ 782us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:59:43.645226 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:59:43.645228 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:59:43.645366 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:59:43.645736 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:59:43.645739 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:59:43.645741 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:59:43.646072 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:59:43.646300 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0924 09:59:43.646615 (+ 315us) server_negotiation.cc:299] Negotiation successful
0924 09:59:43.646786 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260924 09:59:43.647845 16748 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 09:59:43.687614 16928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:59:43.687054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45337 (local address 127.0.0.1:44580)
0924 09:59:43.687332 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0924 09:59:43.687449 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45337: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260924 09:59:43.688673 16928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:59:43.688265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45337 (local address 127.0.0.1:44586)
0924 09:59:43.688420 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0924 09:59:43.688503 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45337: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260924 09:59:43.749223 16928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:59:43.748708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45337 (local address 127.0.0.1:44590)
0924 09:59:43.748935 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0924 09:59:43.749056 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45337: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260924 09:59:43.750202 16928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:59:43.749840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45337 (local address 127.0.0.1:44598)
0924 09:59:43.749970 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0924 09:59:43.750049 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45337: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260924 09:59:43.758543 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu with pid 16625
I20260924 09:59:43.773644 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu pid 16756
I20260924 09:59:43.786228 16899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:59:43.774345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55668 (local address 127.0.97.3:39627)
0924 09:59:43.774521 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0924 09:59:43.774527 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 09:59:43.774546 (+ 19us) server_negotiation.cc:407] Connection header received
0924 09:59:43.774688 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 09:59:43.774693 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 09:59:43.774772 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 09:59:43.774875 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0924 09:59:43.776163 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:59:43.776883 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:59:43.778176 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 09:59:43.778564 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 09:59:43.778671 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 09:59:43.781170 (+ 2499us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 09:59:43.781184 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 09:59:43.781200 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 09:59:43.781231 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 09:59:43.783934 (+ 2703us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:59:43.784602 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:59:43.784609 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:59:43.784612 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:59:43.784749 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 09:59:43.785022 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 09:59:43.785025 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 09:59:43.785027 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 09:59:43.785336 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 09:59:43.785535 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0924 09:59:43.785726 (+ 191us) server_negotiation.cc:299] Negotiation successful
0924 09:59:43.785889 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 09:59:43.786836 16879 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 09:59:43.830643 16928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:59:43.830118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45337 (local address 127.0.0.1:44602)
0924 09:59:43.830339 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0924 09:59:43.830455 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45337: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260924 09:59:43.831780 16928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 09:59:43.831346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45337 (local address 127.0.0.1:44608)
0924 09:59:43.831536 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0924 09:59:43.831622 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45337: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260924 09:59:43.874406 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu with pid 16756
I20260924 09:59:43.887399 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu pid 16403
I20260924 09:59:43.888476 16466 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 09:59:43.999611 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu with pid 16403
I20260924 10:00:04.017460 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
/tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35771
--webserver_interface=127.0.97.62
--webserver_port=40071
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:35771
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:00:04.326844 16932 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:00:04.327116 16932 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:00:04.327203 16932 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:00:04.334290 16932 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260924 10:00:04.334390 16932 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260924 10:00:04.334426 16932 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260924 10:00:04.334475 16932 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:00:04.334530 16932 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:00:04.334594 16932 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:00:04.334636 16932 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:00:04.334686 16932 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:00:04.344288 16932 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:35771
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35771
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=40071
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:00:04.345496 16932 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:00:04.346737 16932 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:00:04.358373 16937 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:00:04.358337 16938 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:00:04.358294 16940 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:00:04.359179 16932 server_base.cc:1034] running on GCE node
I20260924 10:00:04.359966 16932 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:00:04.361336 16932 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:00:04.362551 16932 hybrid_clock.cc:648] HybridClock initialized: now 1790244004362464 us; error 60 us; skew 500 ppm
I20260924 10:00:04.362897 16932 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:00:04 dist-test-slave-w8d2 krb5kdc[15852](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244004, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:00:04.366089 16932 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260924 10:00:04.367579 16932 webserver.cc:460] Webserver started at http://127.0.97.62:40071/ using document root <none> and password file <none>
I20260924 10:00:04.368273 16932 fs_manager.cc:359] Metadata directory not provided
I20260924 10:00:04.368371 16932 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:00:04.373658 16932 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.002s
I20260924 10:00:04.376452 16947 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:00:04.377700 16932 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 10:00:04.377864 16932 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/minicluster-data/master-0/wal
uuid: "6795636462c1442db8265257ea02ed3e"
format_stamp: "Formatted at 2026-09-24 09:59:18 on dist-test-slave-w8d2"
server_key: "ffcf085de30b7e372d06e0619e30efdc"
server_key_iv: "4fbe7fbf78fb40a8cae744590584df5c"
server_key_version: "encryptionkey@0"
I20260924 10:00:04.378796 16932 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:00:04.396327 16932 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:00:04.400213 16932 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:00:04.400461 16932 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:00:04.417326 16932 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:35771
I20260924 10:00:04.417361 16998 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:35771 every 8 connection(s)
I20260924 10:00:04.418787 16932 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/minicluster-data/master-0/data/info.pb
I20260924 10:00:04.425695 16999 sys_catalog.cc:263] Verifying existing consensus state
I20260924 10:00:04.427106 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu as pid 16932
I20260924 10:00:04.428043 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
/tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:45337
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:35771
--webserver_port=44043
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 10:00:04.430524 16999 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6795636462c1442db8265257ea02ed3e: Bootstrap starting.
I20260924 10:00:04.457952 16999 log.cc:824] T 00000000000000000000000000000000 P 6795636462c1442db8265257ea02ed3e: Log is configured to *not* fsync() on all Append() calls
I20260924 10:00:04.480734 16999 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6795636462c1442db8265257ea02ed3e: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 10:00:04.481225 16999 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6795636462c1442db8265257ea02ed3e: Bootstrap complete.
I20260924 10:00:04.490079 16999 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6795636462c1442db8265257ea02ed3e [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6795636462c1442db8265257ea02ed3e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35771 } }
I20260924 10:00:04.491088 16999 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6795636462c1442db8265257ea02ed3e [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6795636462c1442db8265257ea02ed3e, State: Initialized, Role: FOLLOWER
I20260924 10:00:04.491699 16999 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6795636462c1442db8265257ea02ed3e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6795636462c1442db8265257ea02ed3e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35771 } }
I20260924 10:00:04.491916 16999 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6795636462c1442db8265257ea02ed3e [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:00:04.492022 16999 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6795636462c1442db8265257ea02ed3e [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:00:04.492174 16999 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6795636462c1442db8265257ea02ed3e [term 2 FOLLOWER]: Advancing to term 3
I20260924 10:00:04.495774 16999 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6795636462c1442db8265257ea02ed3e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6795636462c1442db8265257ea02ed3e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35771 } }
I20260924 10:00:04.496224 16999 leader_election.cc:302] T 00000000000000000000000000000000 P 6795636462c1442db8265257ea02ed3e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6795636462c1442db8265257ea02ed3e; no voters:
I20260924 10:00:04.496796 16999 leader_election.cc:288] T 00000000000000000000000000000000 P 6795636462c1442db8265257ea02ed3e [CANDIDATE]: Term 3 election: Requested vote from peers
I20260924 10:00:04.496974 17003 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6795636462c1442db8265257ea02ed3e [term 3 FOLLOWER]: Leader election won for term 3
I20260924 10:00:04.497979 17003 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6795636462c1442db8265257ea02ed3e [term 3 LEADER]: Becoming Leader. State: Replica: 6795636462c1442db8265257ea02ed3e, State: Running, Role: LEADER
I20260924 10:00:04.498515 17003 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6795636462c1442db8265257ea02ed3e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6795636462c1442db8265257ea02ed3e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35771 } }
I20260924 10:00:04.498805 16999 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6795636462c1442db8265257ea02ed3e [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:00:04.500939 17005 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6795636462c1442db8265257ea02ed3e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6795636462c1442db8265257ea02ed3e. Latest consensus state: current_term: 3 leader_uuid: "6795636462c1442db8265257ea02ed3e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6795636462c1442db8265257ea02ed3e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35771 } } }
I20260924 10:00:04.501327 17005 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6795636462c1442db8265257ea02ed3e [sys.catalog]: This master's current role is: LEADER
I20260924 10:00:04.501518 17004 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6795636462c1442db8265257ea02ed3e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "6795636462c1442db8265257ea02ed3e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6795636462c1442db8265257ea02ed3e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35771 } } }
I20260924 10:00:04.501817 17004 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6795636462c1442db8265257ea02ed3e [sys.catalog]: This master's current role is: LEADER
I20260924 10:00:04.512236 17012 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:00:04.520402 17012 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:00:04.521888 17012 catalog_manager.cc:1259] Loaded cluster ID: cd7c8363bac24ed7aa68f8264f131800
I20260924 10:00:04.521991 17012 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:00:04.525955 17012 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:00:04.527495 17012 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 6795636462c1442db8265257ea02ed3e: Loaded TSK: 0
I20260924 10:00:04.527637 17012 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 6795636462c1442db8265257ea02ed3e: Loaded TSK: 1
F20260924 10:00:04.528708 17012 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fb582b109ad google::LogMessage::Fail() at ??:0
I20260924 10:00:04.532228 17020 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
@ 0x7fb582b1388a google::LogMessage::SendToLog() at ??:0
@ 0x7fb582b104de google::LogMessage::Flush() at ??:0
@ 0x7fb582b11a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fb5868747d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7fb58686373e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7fb5985431c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7fb598542783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7fb5985ce2a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7fb5985ce0c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7fb5986185e9 std::function<>::operator()() at ??:0
@ 0x7fb598546cdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7fb598544c36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fb5985cc610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7fb5985cc439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fb58575c345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fb58575f91c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7fb58575f799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fb58573aeaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7fb588a146db start_thread at ??:0
@ 0x7fb5819aa61f clone at ??:0
W20260924 10:00:04.811297 17001 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:00:04.811566 17001 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:00:04.811671 17001 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:00:04.818691 17001 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:00:04.818786 17001 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:00:04.818902 17001 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260924 10:00:04.830176 17001 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:45337
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=44043
--tserver_master_addrs=127.0.97.62:35771
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:00:04.831621 17001 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:00:04.832887 17001 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:00:04.845029 17046 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:00:04.844909 17047 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:00:04.845062 17049 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:00:04.851147 17001 server_base.cc:1034] running on GCE node
I20260924 10:00:04.851812 17001 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:00:04.852700 17001 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:00:04.853896 17001 hybrid_clock.cc:648] HybridClock initialized: now 1790244004853842 us; error 46 us; skew 500 ppm
I20260924 10:00:04.854215 17001 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:00:04 dist-test-slave-w8d2 krb5kdc[15852](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244004, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:00:04.857124 17001 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260924 10:00:04.858546 17001 webserver.cc:460] Webserver started at http://127.0.97.1:44043/ using document root <none> and password file <none>
I20260924 10:00:04.859279 17001 fs_manager.cc:359] Metadata directory not provided
I20260924 10:00:04.859395 17001 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:00:04.864513 17001 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 10:00:04.867200 17056 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:00:04.868463 17001 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 10:00:04.868598 17001 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/minicluster-data/ts-0/wal
uuid: "745a6616fac446f39b8f5ee7238814c0"
format_stamp: "Formatted at 2026-09-24 09:59:19 on dist-test-slave-w8d2"
server_key: "c3fb61bd489884f59220a26435b8283e"
server_key_iv: "d4548006e18d18dadb99a6ed0dd10a78"
server_key_version: "encryptionkey@0"
I20260924 10:00:04.869485 17001 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:00:04.897758 17001 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:00:04.901474 17001 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:00:04.901731 17001 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:00:04.902647 17001 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:00:04.904310 17063 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260924 10:00:04.909624 17001 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260924 10:00:04.909735 17001 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.001s
I20260924 10:00:04.909850 17001 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260924 10:00:04.912983 17001 ts_tablet_manager.cc:610] Registered 1 tablets
I20260924 10:00:04.913072 17001 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.001s sys 0.001s
I20260924 10:00:04.913447 17063 tablet_bootstrap.cc:492] T 0443f524c3334c78953dbeb19f2a700a P 745a6616fac446f39b8f5ee7238814c0: Bootstrap starting.
I20260924 10:00:04.943867 17063 log.cc:824] T 0443f524c3334c78953dbeb19f2a700a P 745a6616fac446f39b8f5ee7238814c0: Log is configured to *not* fsync() on all Append() calls
I20260924 10:00:04.946862 17001 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:45337
I20260924 10:00:04.946870 17170 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:45337 every 8 connection(s)
I20260924 10:00:04.948403 17001 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/minicluster-data/ts-0/data/info.pb
I20260924 10:00:04.953130 17173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:04.951116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35771 (local address 127.0.97.1:52719)
0924 10:00:04.951957 (+ 841us) negotiation.cc:107] Waiting for socket to connect
0924 10:00:04.952200 (+ 243us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":605,"thread_start_us":369,"threads_started":1}
W20260924 10:00:04.953663 17171 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:35771 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:35771: Client connection negotiation failed: client connection to 127.0.97.62:35771: connect: Connection refused (error 111)
I20260924 10:00:04.954413 17173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:04.954125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35771 (local address 127.0.97.1:52163)
0924 10:00:04.954250 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0924 10:00:04.954312 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260924 10:00:04.955175 17173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:04.954868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35771 (local address 127.0.97.1:53805)
0924 10:00:04.954991 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0924 10:00:04.955062 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260924 10:00:04.955413 17171 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 10:00:04.955988 17063 tablet_bootstrap.cc:492] T 0443f524c3334c78953dbeb19f2a700a P 745a6616fac446f39b8f5ee7238814c0: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 10:00:04.956441 17063 tablet_bootstrap.cc:492] T 0443f524c3334c78953dbeb19f2a700a P 745a6616fac446f39b8f5ee7238814c0: Bootstrap complete.
I20260924 10:00:04.957455 17063 ts_tablet_manager.cc:1397] T 0443f524c3334c78953dbeb19f2a700a P 745a6616fac446f39b8f5ee7238814c0: Time spent bootstrapping tablet: real 0.044s user 0.021s sys 0.021s
I20260924 10:00:04.958738 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu as pid 17001
I20260924 10:00:04.959348 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
/tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:40543
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:35771
--webserver_port=45649
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 10:00:04.965651 17063 raft_consensus.cc:348] T 0443f524c3334c78953dbeb19f2a700a P 745a6616fac446f39b8f5ee7238814c0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745a6616fac446f39b8f5ee7238814c0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45337 } } peers { permanent_uuid: "f3295c0ca2c54aea8a83fe6cbfb8bc14" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39627 } } peers { permanent_uuid: "e785aaa5a6044212b18b8397ce8483e0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40543 } }
I20260924 10:00:04.966609 17063 raft_consensus.cc:729] T 0443f524c3334c78953dbeb19f2a700a P 745a6616fac446f39b8f5ee7238814c0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 745a6616fac446f39b8f5ee7238814c0, State: Initialized, Role: FOLLOWER
I20260924 10:00:04.967281 17063 consensus_queue.cc:260] T 0443f524c3334c78953dbeb19f2a700a P 745a6616fac446f39b8f5ee7238814c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745a6616fac446f39b8f5ee7238814c0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45337 } } peers { permanent_uuid: "f3295c0ca2c54aea8a83fe6cbfb8bc14" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39627 } } peers { permanent_uuid: "e785aaa5a6044212b18b8397ce8483e0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40543 } }
I20260924 10:00:04.968843 17173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:04.968491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35771 (local address 127.0.97.1:59167)
0924 10:00:04.968652 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0924 10:00:04.968726 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260924 10:00:04.968921 17063 ts_tablet_manager.cc:1428] T 0443f524c3334c78953dbeb19f2a700a P 745a6616fac446f39b8f5ee7238814c0: Time spent starting tablet: real 0.011s user 0.008s sys 0.002s
I20260924 10:00:04.969751 17173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:04.969447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35771 (local address 127.0.97.1:59029)
0924 10:00:04.969581 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0924 10:00:04.969644 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260924 10:00:05.273381 17174 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:00:05.273667 17174 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:00:05.273809 17174 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:00:05.280889 17174 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:00:05.281003 17174 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:00:05.281157 17174 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260924 10:00:05.290512 17174 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:40543
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=45649
--tserver_master_addrs=127.0.97.62:35771
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:00:05.291749 17174 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:00:05.293001 17174 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:00:05.305085 17183 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:00:05.305146 17180 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:00:05.305107 17181 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:00:05.305850 17174 server_base.cc:1034] running on GCE node
I20260924 10:00:05.306393 17174 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:00:05.307300 17174 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:00:05.308497 17174 hybrid_clock.cc:648] HybridClock initialized: now 1790244005308448 us; error 40 us; skew 500 ppm
I20260924 10:00:05.308802 17174 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:00:05 dist-test-slave-w8d2 krb5kdc[15852](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244005, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:00:05.311666 17174 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260924 10:00:05.313054 17174 webserver.cc:460] Webserver started at http://127.0.97.2:45649/ using document root <none> and password file <none>
I20260924 10:00:05.313686 17174 fs_manager.cc:359] Metadata directory not provided
I20260924 10:00:05.313808 17174 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:00:05.318758 17174 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260924 10:00:05.321339 17190 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:00:05.322598 17174 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 10:00:05.322746 17174 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/minicluster-data/ts-1/wal
uuid: "e785aaa5a6044212b18b8397ce8483e0"
format_stamp: "Formatted at 2026-09-24 09:59:19 on dist-test-slave-w8d2"
server_key: "0fe5933f754a5200bd6a18cbad61fa62"
server_key_iv: "7b4dd5fbe13dbc0c63f092cfab0b95ef"
server_key_version: "encryptionkey@0"
I20260924 10:00:05.323766 17174 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:00:05.335986 17174 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:00:05.339669 17174 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:00:05.339926 17174 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:00:05.340847 17174 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:00:05.342168 17174 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:00:05.342262 17174 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:00:05.342371 17174 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:00:05.342438 17174 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:00:05.372953 17174 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:40543
I20260924 10:00:05.373037 17302 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:40543 every 8 connection(s)
I20260924 10:00:05.374430 17174 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/minicluster-data/ts-1/data/info.pb
I20260924 10:00:05.379317 17305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:05.377244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35771 (local address 127.0.97.2:46497)
0924 10:00:05.377951 (+ 707us) negotiation.cc:107] Waiting for socket to connect
0924 10:00:05.378139 (+ 188us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":264,"threads_started":1}
W20260924 10:00:05.380096 17303 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:35771 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:35771: Client connection negotiation failed: client connection to 127.0.97.62:35771: connect: Connection refused (error 111)
I20260924 10:00:05.380915 17305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:05.380576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35771 (local address 127.0.97.2:52637)
0924 10:00:05.380739 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0924 10:00:05.380809 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260924 10:00:05.381448 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu as pid 17174
I20260924 10:00:05.381915 17305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:05.381494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35771 (local address 127.0.97.2:47189)
0924 10:00:05.381699 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0924 10:00:05.381797 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"mutex_wait_us":77}
I20260924 10:00:05.382107 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
/tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:39627
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:35771
--webserver_port=46083
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:00:05.382139 17303 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 10:00:05.689150 17306 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:00:05.689430 17306 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:00:05.689550 17306 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:00:05.696609 17306 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:00:05.696729 17306 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:00:05.696883 17306 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260924 10:00:05.706350 17306 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:39627
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=46083
--tserver_master_addrs=127.0.97.62:35771
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:00:05.707609 17306 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:00:05.708848 17306 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:00:05.721230 17314 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:00:05.721331 17311 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:00:05.721311 17312 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:00:05.722018 17306 server_base.cc:1034] running on GCE node
I20260924 10:00:05.722584 17306 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:00:05.723536 17306 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:00:05.724715 17306 hybrid_clock.cc:648] HybridClock initialized: now 1790244005724665 us; error 48 us; skew 500 ppm
I20260924 10:00:05.724994 17306 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:00:05 dist-test-slave-w8d2 krb5kdc[15852](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244005, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:00:05.727911 17306 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260924 10:00:05.729317 17306 webserver.cc:460] Webserver started at http://127.0.97.3:46083/ using document root <none> and password file <none>
I20260924 10:00:05.729930 17306 fs_manager.cc:359] Metadata directory not provided
I20260924 10:00:05.730027 17306 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:00:05.735493 17306 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260924 10:00:05.738056 17321 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:00:05.739372 17306 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 10:00:05.739513 17306 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/minicluster-data/ts-2/wal
uuid: "f3295c0ca2c54aea8a83fe6cbfb8bc14"
format_stamp: "Formatted at 2026-09-24 09:59:20 on dist-test-slave-w8d2"
server_key: "336612d680ec5c24a7155b536bfeebd2"
server_key_iv: "999a87a783ef1800aab3eeb7bb26335f"
server_key_version: "encryptionkey@0"
I20260924 10:00:05.740442 17306 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:00:05.766216 17306 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:00:05.770176 17306 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:00:05.770423 17306 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:00:05.771377 17306 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:00:05.772658 17306 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:00:05.772740 17306 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:00:05.772840 17306 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:00:05.772913 17306 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:00:05.801366 17306 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:39627
I20260924 10:00:05.801448 17433 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:39627 every 8 connection(s)
I20260924 10:00:05.802726 17306 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790243735634835-388-0/minicluster-data/ts-2/data/info.pb
I20260924 10:00:05.807129 17436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:05.805440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35771 (local address 127.0.97.3:57119)
0924 10:00:05.806103 (+ 663us) negotiation.cc:107] Waiting for socket to connect
0924 10:00:05.806298 (+ 195us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":252,"threads_started":1}
W20260924 10:00:05.807662 17434 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:35771 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:35771: Client connection negotiation failed: client connection to 127.0.97.62:35771: connect: Connection refused (error 111)
I20260924 10:00:05.808348 17436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:05.808071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35771 (local address 127.0.97.3:50323)
0924 10:00:05.808193 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0924 10:00:05.808253 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260924 10:00:05.809015 17436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:05.808755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35771 (local address 127.0.97.3:57163)
0924 10:00:05.808853 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0924 10:00:05.808919 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260924 10:00:05.809207 17434 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 10:00:05.811748 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu as pid 17306
I20260924 10:00:05.971359 17438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:05.970575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35771 (local address 127.0.97.1:48869)
0924 10:00:05.971018 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0924 10:00:05.971122 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":207,"threads_started":1}
I20260924 10:00:06.383553 17439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:06.382763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35771 (local address 127.0.97.2:42809)
0924 10:00:06.383209 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0924 10:00:06.383329 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":207,"threads_started":1}
I20260924 10:00:06.534967 17440 raft_consensus.cc:482] T 0443f524c3334c78953dbeb19f2a700a P 745a6616fac446f39b8f5ee7238814c0 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 10:00:06.535313 17440 raft_consensus.cc:504] T 0443f524c3334c78953dbeb19f2a700a P 745a6616fac446f39b8f5ee7238814c0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745a6616fac446f39b8f5ee7238814c0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45337 } } peers { permanent_uuid: "f3295c0ca2c54aea8a83fe6cbfb8bc14" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39627 } } peers { permanent_uuid: "e785aaa5a6044212b18b8397ce8483e0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40543 } }
I20260924 10:00:06.536816 17440 leader_election.cc:288] T 0443f524c3334c78953dbeb19f2a700a P 745a6616fac446f39b8f5ee7238814c0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f3295c0ca2c54aea8a83fe6cbfb8bc14 (127.0.97.3:39627), e785aaa5a6044212b18b8397ce8483e0 (127.0.97.2:40543)
Sep 24 10:00:06 dist-test-slave-w8d2 krb5kdc[15852](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244004, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
Sep 24 10:00:06 dist-test-slave-w8d2 krb5kdc[15852](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244004, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260924 10:00:06.576803 17442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:06.537231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:39627 (local address 127.0.97.1:60779)
0924 10:00:06.552369 (+ 15138us) negotiation.cc:107] Waiting for socket to connect
0924 10:00:06.552383 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 10:00:06.552924 (+ 541us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:00:06.557075 (+ 4151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:00:06.557087 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:00:06.558244 (+ 1157us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:00:06.559391 (+ 1147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:00:06.559421 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:00:06.561674 (+ 2253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:00:06.561692 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:00:06.563996 (+ 2304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:00:06.564012 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:00:06.565668 (+ 1656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:00:06.565674 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:00:06.566044 (+ 370us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:00:06.566897 (+ 853us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:00:06.566936 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:00:06.570148 (+ 3212us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:00:06.573420 (+ 3272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:00:06.573426 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:00:06.573430 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:00:06.573933 (+ 503us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:00:06.574329 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:00:06.574334 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:00:06.574338 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:00:06.574548 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:00:06.575299 (+ 751us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:00:06.575306 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:00:06.575625 (+ 319us) client_negotiation.cc:769] Sending connection context
0924 10:00:06.575845 (+ 220us) client_negotiation.cc:240] Negotiation successful
0924 10:00:06.576362 (+ 517us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15041,"mutex_wait_us":384,"spinlock_wait_cycles":50432,"thread_start_us":14713,"threads_started":1}
I20260924 10:00:06.576812 17444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:06.537560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:60779 (local address 127.0.97.3:39627)
0924 10:00:06.553328 (+ 15768us) server_negotiation.cc:206] Beginning negotiation
0924 10:00:06.553344 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 10:00:06.553390 (+ 46us) server_negotiation.cc:407] Connection header received
0924 10:00:06.555992 (+ 2602us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:00:06.556002 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:00:06.556518 (+ 516us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:00:06.556891 (+ 373us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:00:06.559599 (+ 2708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:00:06.561460 (+ 1861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:00:06.564133 (+ 2673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:00:06.565171 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:00:06.565352 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:00:06.570440 (+ 5088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:00:06.570446 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:00:06.570459 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:00:06.570500 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:00:06.573266 (+ 2766us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:00:06.574064 (+ 798us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:00:06.574071 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:00:06.574077 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:00:06.574229 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:00:06.574653 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:00:06.574656 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:00:06.574658 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:00:06.575141 (+ 483us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:00:06.575338 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0924 10:00:06.576096 (+ 758us) server_negotiation.cc:299] Negotiation successful
0924 10:00:06.576447 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15588,"thread_start_us":15474,"threads_started":1}
I20260924 10:00:06.576792 17443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:06.537320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:35985 (local address 127.0.97.2:40543)
0924 10:00:06.553719 (+ 16399us) server_negotiation.cc:206] Beginning negotiation
0924 10:00:06.553733 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 10:00:06.553774 (+ 41us) server_negotiation.cc:407] Connection header received
0924 10:00:06.555989 (+ 2215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:00:06.556002 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:00:06.556518 (+ 516us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:00:06.556891 (+ 373us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:00:06.559599 (+ 2708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:00:06.561460 (+ 1861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:00:06.564133 (+ 2673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:00:06.565193 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:00:06.565353 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:00:06.570126 (+ 4773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:00:06.570138 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:00:06.570159 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:00:06.570214 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:00:06.573252 (+ 3038us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:00:06.574085 (+ 833us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:00:06.574091 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:00:06.574097 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:00:06.574236 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:00:06.574658 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:00:06.574661 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:00:06.574663 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:00:06.575155 (+ 492us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:00:06.575349 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0924 10:00:06.576079 (+ 730us) server_negotiation.cc:299] Negotiation successful
0924 10:00:06.576424 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16194,"thread_start_us":17109,"threads_started":1}
I20260924 10:00:06.577500 17441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:06.537176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:40543 (local address 127.0.97.1:35985)
0924 10:00:06.552210 (+ 15034us) negotiation.cc:107] Waiting for socket to connect
0924 10:00:06.552278 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0924 10:00:06.553070 (+ 792us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:00:06.557075 (+ 4005us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:00:06.557087 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:00:06.558243 (+ 1156us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:00:06.559391 (+ 1148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:00:06.559421 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:00:06.561688 (+ 2267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:00:06.561692 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:00:06.563996 (+ 2304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:00:06.564012 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:00:06.565679 (+ 1667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:00:06.565685 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:00:06.566048 (+ 363us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:00:06.566900 (+ 852us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:00:06.566937 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:00:06.569756 (+ 2819us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:00:06.573401 (+ 3645us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:00:06.573409 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:00:06.573424 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:00:06.573953 (+ 529us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:00:06.574339 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:00:06.574344 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:00:06.574348 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:00:06.574549 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:00:06.575300 (+ 751us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:00:06.575308 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:00:06.575621 (+ 313us) client_negotiation.cc:769] Sending connection context
0924 10:00:06.575844 (+ 223us) client_negotiation.cc:240] Negotiation successful
0924 10:00:06.577146 (+ 1302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14882,"mutex_wait_us":1081,"spinlock_wait_cycles":78848,"thread_start_us":15676,"threads_started":1}
W20260924 10:00:06.580197 17060 leader_election.cc:341] T 0443f524c3334c78953dbeb19f2a700a P 745a6616fac446f39b8f5ee7238814c0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e785aaa5a6044212b18b8397ce8483e0 (127.0.97.2:40543): Not found: Tablet not found: 0443f524c3334c78953dbeb19f2a700a
W20260924 10:00:06.580466 17058 leader_election.cc:341] T 0443f524c3334c78953dbeb19f2a700a P 745a6616fac446f39b8f5ee7238814c0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f3295c0ca2c54aea8a83fe6cbfb8bc14 (127.0.97.3:39627): Not found: Tablet not found: 0443f524c3334c78953dbeb19f2a700a
I20260924 10:00:06.580586 17058 leader_election.cc:302] T 0443f524c3334c78953dbeb19f2a700a P 745a6616fac446f39b8f5ee7238814c0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 745a6616fac446f39b8f5ee7238814c0; no voters: e785aaa5a6044212b18b8397ce8483e0, f3295c0ca2c54aea8a83fe6cbfb8bc14
I20260924 10:00:06.580971 17440 raft_consensus.cc:2726] T 0443f524c3334c78953dbeb19f2a700a P 745a6616fac446f39b8f5ee7238814c0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 10:00:06.810617 17445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:06.809705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35771 (local address 127.0.97.3:53493)
0924 10:00:06.810181 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0924 10:00:06.810305 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":223,"threads_started":1}
I20260924 10:00:06.972690 17441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:06.972264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35771 (local address 127.0.97.1:50017)
0924 10:00:06.972424 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0924 10:00:06.972529 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260924 10:00:07.385073 17446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:07.384419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35771 (local address 127.0.97.2:45253)
0924 10:00:07.384835 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0924 10:00:07.384923 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":224,"threads_started":1}
I20260924 10:00:07.812215 17447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:07.811542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35771 (local address 127.0.97.3:54565)
0924 10:00:07.811954 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0924 10:00:07.812036 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":197,"threads_started":1}
I20260924 10:00:07.974366 17448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:07.973595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35771 (local address 127.0.97.1:33047)
0924 10:00:07.974104 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0924 10:00:07.974186 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":224,"threads_started":1}
I20260924 10:00:08.386634 17449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:08.385925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35771 (local address 127.0.97.2:58479)
0924 10:00:08.386388 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0924 10:00:08.386471 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":280,"threads_started":1}
I20260924 10:00:08.473371 17450 raft_consensus.cc:482] T 0443f524c3334c78953dbeb19f2a700a P 745a6616fac446f39b8f5ee7238814c0 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 10:00:08.473560 17450 raft_consensus.cc:504] T 0443f524c3334c78953dbeb19f2a700a P 745a6616fac446f39b8f5ee7238814c0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745a6616fac446f39b8f5ee7238814c0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45337 } } peers { permanent_uuid: "f3295c0ca2c54aea8a83fe6cbfb8bc14" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39627 } } peers { permanent_uuid: "e785aaa5a6044212b18b8397ce8483e0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40543 } }
I20260924 10:00:08.474195 17450 leader_election.cc:288] T 0443f524c3334c78953dbeb19f2a700a P 745a6616fac446f39b8f5ee7238814c0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f3295c0ca2c54aea8a83fe6cbfb8bc14 (127.0.97.3:39627), e785aaa5a6044212b18b8397ce8483e0 (127.0.97.2:40543)
W20260924 10:00:08.475164 17060 leader_election.cc:341] T 0443f524c3334c78953dbeb19f2a700a P 745a6616fac446f39b8f5ee7238814c0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e785aaa5a6044212b18b8397ce8483e0 (127.0.97.2:40543): Not found: Tablet not found: 0443f524c3334c78953dbeb19f2a700a
W20260924 10:00:08.475338 17058 leader_election.cc:341] T 0443f524c3334c78953dbeb19f2a700a P 745a6616fac446f39b8f5ee7238814c0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f3295c0ca2c54aea8a83fe6cbfb8bc14 (127.0.97.3:39627): Not found: Tablet not found: 0443f524c3334c78953dbeb19f2a700a
I20260924 10:00:08.475461 17058 leader_election.cc:302] T 0443f524c3334c78953dbeb19f2a700a P 745a6616fac446f39b8f5ee7238814c0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 745a6616fac446f39b8f5ee7238814c0; no voters: e785aaa5a6044212b18b8397ce8483e0, f3295c0ca2c54aea8a83fe6cbfb8bc14
I20260924 10:00:08.475695 17450 raft_consensus.cc:2726] T 0443f524c3334c78953dbeb19f2a700a P 745a6616fac446f39b8f5ee7238814c0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 10:00:08.813635 17451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:08.813014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35771 (local address 127.0.97.3:59141)
0924 10:00:08.813409 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0924 10:00:08.813483 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":200,"threads_started":1}
I20260924 10:00:08.975905 17452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:08.975179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35771 (local address 127.0.97.1:35079)
0924 10:00:08.975648 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0924 10:00:08.975725 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":259,"threads_started":1}
I20260924 10:00:09.388159 17453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:09.387505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35771 (local address 127.0.97.2:34321)
0924 10:00:09.387891 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0924 10:00:09.387972 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":188,"threads_started":1}
I20260924 10:00:09.815001 17454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:09.814357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35771 (local address 127.0.97.3:45599)
0924 10:00:09.814746 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0924 10:00:09.814827 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":216,"threads_started":1}
I20260924 10:00:09.977743 17455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:09.977048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35771 (local address 127.0.97.1:35803)
0924 10:00:09.977477 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0924 10:00:09.977568 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":215,"threads_started":1}
I20260924 10:00:10.389802 17456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:10.389048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35771 (local address 127.0.97.2:59105)
0924 10:00:10.389544 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0924 10:00:10.389620 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":296,"threads_started":1}
I20260924 10:00:10.816512 17457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:10.815805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35771 (local address 127.0.97.3:46859)
0924 10:00:10.816277 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0924 10:00:10.816358 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":204,"threads_started":1}
I20260924 10:00:10.979357 17458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:10.978687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35771 (local address 127.0.97.1:40425)
0924 10:00:10.979081 (+ 394us) negotiation.cc:107] Waiting for socket to connect
0924 10:00:10.979163 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":200,"threads_started":1}
I20260924 10:00:11.030798 17459 raft_consensus.cc:482] T 0443f524c3334c78953dbeb19f2a700a P 745a6616fac446f39b8f5ee7238814c0 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 10:00:11.030984 17459 raft_consensus.cc:504] T 0443f524c3334c78953dbeb19f2a700a P 745a6616fac446f39b8f5ee7238814c0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745a6616fac446f39b8f5ee7238814c0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45337 } } peers { permanent_uuid: "f3295c0ca2c54aea8a83fe6cbfb8bc14" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39627 } } peers { permanent_uuid: "e785aaa5a6044212b18b8397ce8483e0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40543 } }
I20260924 10:00:11.031646 17459 leader_election.cc:288] T 0443f524c3334c78953dbeb19f2a700a P 745a6616fac446f39b8f5ee7238814c0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f3295c0ca2c54aea8a83fe6cbfb8bc14 (127.0.97.3:39627), e785aaa5a6044212b18b8397ce8483e0 (127.0.97.2:40543)
W20260924 10:00:11.032490 17060 leader_election.cc:341] T 0443f524c3334c78953dbeb19f2a700a P 745a6616fac446f39b8f5ee7238814c0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e785aaa5a6044212b18b8397ce8483e0 (127.0.97.2:40543): Not found: Tablet not found: 0443f524c3334c78953dbeb19f2a700a
W20260924 10:00:11.032660 17058 leader_election.cc:341] T 0443f524c3334c78953dbeb19f2a700a P 745a6616fac446f39b8f5ee7238814c0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f3295c0ca2c54aea8a83fe6cbfb8bc14 (127.0.97.3:39627): Not found: Tablet not found: 0443f524c3334c78953dbeb19f2a700a
I20260924 10:00:11.032753 17058 leader_election.cc:302] T 0443f524c3334c78953dbeb19f2a700a P 745a6616fac446f39b8f5ee7238814c0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 745a6616fac446f39b8f5ee7238814c0; no voters: e785aaa5a6044212b18b8397ce8483e0, f3295c0ca2c54aea8a83fe6cbfb8bc14
I20260924 10:00:11.032958 17459 raft_consensus.cc:2726] T 0443f524c3334c78953dbeb19f2a700a P 745a6616fac446f39b8f5ee7238814c0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 10:00:11.391450 17460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:11.390687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35771 (local address 127.0.97.2:53389)
0924 10:00:11.391148 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0924 10:00:11.391267 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":242,"threads_started":1}
I20260924 10:00:11.818020 17461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:11.817316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35771 (local address 127.0.97.3:42049)
0924 10:00:11.817772 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0924 10:00:11.817852 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":227,"threads_started":1}
I20260924 10:00:11.980937 17462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:11.980144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35771 (local address 127.0.97.1:48431)
0924 10:00:11.980639 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0924 10:00:11.980722 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":282,"threads_started":1}
I20260924 10:00:12.393060 17463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:12.392253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35771 (local address 127.0.97.2:56371)
0924 10:00:12.392760 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0924 10:00:12.392838 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":219,"threads_started":1}
I20260924 10:00:12.819808 17464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:12.819006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35771 (local address 127.0.97.3:50999)
0924 10:00:12.819542 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0924 10:00:12.819626 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":268,"threads_started":1}
I20260924 10:00:12.982707 17465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:12.981925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35771 (local address 127.0.97.1:59901)
0924 10:00:12.982441 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0924 10:00:12.982549 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":266,"threads_started":1}
I20260924 10:00:13.394753 17466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:13.393995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35771 (local address 127.0.97.2:36821)
0924 10:00:13.394483 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0924 10:00:13.394584 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":248,"threads_started":1}
I20260924 10:00:13.821628 17467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:13.820813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35771 (local address 127.0.97.3:45645)
0924 10:00:13.821301 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0924 10:00:13.821400 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":283,"threads_started":1}
I20260924 10:00:13.984524 17468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:13.983761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35771 (local address 127.0.97.1:39277)
0924 10:00:13.984241 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0924 10:00:13.984333 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":237,"threads_started":1}
I20260924 10:00:14.396579 17469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:14.395822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35771 (local address 127.0.97.2:58289)
0924 10:00:14.396322 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0924 10:00:14.396417 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":252,"threads_started":1}
I20260924 10:00:14.823321 17470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:14.822540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35771 (local address 127.0.97.3:38439)
0924 10:00:14.823044 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0924 10:00:14.823122 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":262,"threads_started":1}
I20260924 10:00:14.986342 17471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:14.985641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35771 (local address 127.0.97.1:42791)
0924 10:00:14.986084 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0924 10:00:14.986175 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":219,"threads_started":1}
I20260924 10:00:15.378726 17472 raft_consensus.cc:482] T 0443f524c3334c78953dbeb19f2a700a P 745a6616fac446f39b8f5ee7238814c0 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 10:00:15.378902 17472 raft_consensus.cc:504] T 0443f524c3334c78953dbeb19f2a700a P 745a6616fac446f39b8f5ee7238814c0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745a6616fac446f39b8f5ee7238814c0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45337 } } peers { permanent_uuid: "f3295c0ca2c54aea8a83fe6cbfb8bc14" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39627 } } peers { permanent_uuid: "e785aaa5a6044212b18b8397ce8483e0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40543 } }
I20260924 10:00:15.379539 17472 leader_election.cc:288] T 0443f524c3334c78953dbeb19f2a700a P 745a6616fac446f39b8f5ee7238814c0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f3295c0ca2c54aea8a83fe6cbfb8bc14 (127.0.97.3:39627), e785aaa5a6044212b18b8397ce8483e0 (127.0.97.2:40543)
W20260924 10:00:15.380419 17060 leader_election.cc:341] T 0443f524c3334c78953dbeb19f2a700a P 745a6616fac446f39b8f5ee7238814c0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e785aaa5a6044212b18b8397ce8483e0 (127.0.97.2:40543): Not found: Tablet not found: 0443f524c3334c78953dbeb19f2a700a
W20260924 10:00:15.380654 17058 leader_election.cc:341] T 0443f524c3334c78953dbeb19f2a700a P 745a6616fac446f39b8f5ee7238814c0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f3295c0ca2c54aea8a83fe6cbfb8bc14 (127.0.97.3:39627): Not found: Tablet not found: 0443f524c3334c78953dbeb19f2a700a
I20260924 10:00:15.380767 17058 leader_election.cc:302] T 0443f524c3334c78953dbeb19f2a700a P 745a6616fac446f39b8f5ee7238814c0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 745a6616fac446f39b8f5ee7238814c0; no voters: e785aaa5a6044212b18b8397ce8483e0, f3295c0ca2c54aea8a83fe6cbfb8bc14
I20260924 10:00:15.380990 17472 raft_consensus.cc:2726] T 0443f524c3334c78953dbeb19f2a700a P 745a6616fac446f39b8f5ee7238814c0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 10:00:15.398080 17473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:15.397401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35771 (local address 127.0.97.2:56727)
0924 10:00:15.397837 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0924 10:00:15.397913 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":202,"threads_started":1}
I20260924 10:00:15.824983 17474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:15.824252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35771 (local address 127.0.97.3:43365)
0924 10:00:15.824707 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0924 10:00:15.824800 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":232,"threads_started":1}
I20260924 10:00:15.987967 17475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:15.987273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35771 (local address 127.0.97.1:54681)
0924 10:00:15.987708 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0924 10:00:15.987790 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":222,"threads_started":1}
I20260924 10:00:16.399773 17476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:16.398952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35771 (local address 127.0.97.2:54093)
0924 10:00:16.399500 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0924 10:00:16.399594 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":305,"threads_started":1}
I20260924 10:00:16.826699 17477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:16.825921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35771 (local address 127.0.97.3:33673)
0924 10:00:16.826450 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0924 10:00:16.826540 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":287,"threads_started":1}
I20260924 10:00:16.989499 17478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:16.988818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35771 (local address 127.0.97.1:53921)
0924 10:00:16.989258 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0924 10:00:16.989341 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":245,"threads_started":1}
I20260924 10:00:17.401367 17479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:17.400613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35771 (local address 127.0.97.2:33201)
0924 10:00:17.401111 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0924 10:00:17.401195 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":252,"threads_started":1}
I20260924 10:00:17.828331 17480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:17.827646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35771 (local address 127.0.97.3:44347)
0924 10:00:17.828083 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0924 10:00:17.828164 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":220,"threads_started":1}
I20260924 10:00:17.991175 17481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:17.990439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35771 (local address 127.0.97.1:44971)
0924 10:00:17.990896 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0924 10:00:17.990980 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":240,"threads_started":1}
I20260924 10:00:18.403035 17482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:18.402278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35771 (local address 127.0.97.2:48559)
0924 10:00:18.402754 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0924 10:00:18.402857 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":229,"threads_started":1}
I20260924 10:00:18.829973 17483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:18.829281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35771 (local address 127.0.97.3:58305)
0924 10:00:18.829737 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0924 10:00:18.829815 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":256,"threads_started":1}
I20260924 10:00:18.992825 17484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:18.992088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35771 (local address 127.0.97.1:38017)
0924 10:00:18.992563 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0924 10:00:18.992642 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":264,"threads_started":1}
I20260924 10:00:19.404573 17485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:19.403873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35771 (local address 127.0.97.2:60569)
0924 10:00:19.404285 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0924 10:00:19.404362 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":204,"threads_started":1}
I20260924 10:00:19.831462 17486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:19.830774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35771 (local address 127.0.97.3:38631)
0924 10:00:19.831169 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0924 10:00:19.831269 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":196,"threads_started":1}
I20260924 10:00:19.994776 17487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:19.993987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35771 (local address 127.0.97.1:55879)
0924 10:00:19.994478 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0924 10:00:19.994577 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":247,"threads_started":1}
I20260924 10:00:20.406443 17488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:20.405750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35771 (local address 127.0.97.2:47583)
0924 10:00:20.406191 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0924 10:00:20.406275 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":220,"threads_started":1}
I20260924 10:00:20.821348 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu pid 17001
I20260924 10:00:20.833086 17490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:20.832417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35771 (local address 127.0.97.3:59941)
0924 10:00:20.832831 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0924 10:00:20.832924 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":208,"threads_started":1}
I20260924 10:00:20.840178 17489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:20.822380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48404 (local address 127.0.97.1:45337)
0924 10:00:20.823076 (+ 696us) server_negotiation.cc:206] Beginning negotiation
0924 10:00:20.823084 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 10:00:20.823111 (+ 27us) server_negotiation.cc:407] Connection header received
0924 10:00:20.823192 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:00:20.823201 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:00:20.823518 (+ 317us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:00:20.823984 (+ 466us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:00:20.825384 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:00:20.826384 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:00:20.827618 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:00:20.828274 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:00:20.828375 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:00:20.835139 (+ 6764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:00:20.835152 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:00:20.835168 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:00:20.835202 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:00:20.837765 (+ 2563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:00:20.838361 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:00:20.838368 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:00:20.838371 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:00:20.838504 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:00:20.838815 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:00:20.838818 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:00:20.838820 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:00:20.839179 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:00:20.839444 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0924 10:00:20.839670 (+ 226us) server_negotiation.cc:299] Negotiation successful
0924 10:00:20.839883 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":529,"thread_start_us":417,"threads_started":1}
I20260924 10:00:20.841342 17166 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:00:20.929195 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu with pid 17001
I20260924 10:00:20.944044 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu pid 17174
I20260924 10:00:20.956255 17492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:20.944827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34716 (local address 127.0.97.2:40543)
0924 10:00:20.945353 (+ 526us) server_negotiation.cc:206] Beginning negotiation
0924 10:00:20.945357 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:00:20.945373 (+ 16us) server_negotiation.cc:407] Connection header received
0924 10:00:20.945427 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:00:20.945431 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:00:20.945488 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:00:20.945574 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:00:20.946808 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:00:20.947635 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:00:20.948757 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:00:20.949160 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:00:20.949251 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:00:20.951769 (+ 2518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:00:20.951773 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:00:20.951777 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:00:20.951808 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:00:20.954129 (+ 2321us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:00:20.954718 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:00:20.954721 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:00:20.954724 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:00:20.954848 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:00:20.955112 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:00:20.955115 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:00:20.955117 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:00:20.955385 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:00:20.955574 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0924 10:00:20.955814 (+ 240us) server_negotiation.cc:299] Negotiation successful
0924 10:00:20.955974 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":431,"thread_start_us":326,"threads_started":1}
I20260924 10:00:20.956954 17298 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:00:21.039894 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu with pid 17174
I20260924 10:00:21.054171 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu pid 17306
I20260924 10:00:21.066404 17494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:21.054866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43422 (local address 127.0.97.3:39627)
0924 10:00:21.055388 (+ 522us) server_negotiation.cc:206] Beginning negotiation
0924 10:00:21.055393 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:00:21.055409 (+ 16us) server_negotiation.cc:407] Connection header received
0924 10:00:21.055462 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:00:21.055466 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:00:21.055534 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:00:21.055618 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:00:21.056856 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:00:21.057581 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:00:21.058760 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:00:21.059152 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:00:21.059337 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:00:21.061601 (+ 2264us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:00:21.061606 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:00:21.061609 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:00:21.061645 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:00:21.064152 (+ 2507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:00:21.064841 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:00:21.064845 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:00:21.064847 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:00:21.064981 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:00:21.065313 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:00:21.065316 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:00:21.065318 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:00:21.065551 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:00:21.065765 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0924 10:00:21.065955 (+ 190us) server_negotiation.cc:299] Negotiation successful
0924 10:00:21.066113 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":410,"thread_start_us":249,"threads_started":1}
I20260924 10:00:21.067000 17429 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:00:21.148165 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu with pid 17306
2026-09-24T10:00:21Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (63846 ms)
[----------] 30 tests from SecurityITest (285472 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790243735634835-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:00:21 dist-test-slave-w8d2 krb5kdc[17498](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:00:21 dist-test-slave-w8d2 krb5kdc[17498](info): set up 2 sockets
Sep 24 10:00:21 dist-test-slave-w8d2 krb5kdc[17498](info): commencing operation
krb5kdc: starting...
W20260924 10:00:22.143013 388 mini_kdc.cc:121] Time spent starting KDC: real 0.960s user 0.003s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790243735634835-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790243735634835-388-0/krb5kdc/test-user.keytab.
Sep 24 10:00:22 dist-test-slave-w8d2 krb5kdc[17498](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244022, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:00:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:00:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790243735634835-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790243735634835-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260924 10:00:22.356849 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
/tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790243735634835-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790243735634835-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790243735634835-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790243735634835-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40793
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790243735634835-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44359
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:40793
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790243735634835-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790243735634835-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790243735634835-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:00:22.664896 17511 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:00:22.665198 17511 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:00:22.665320 17511 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:00:22.672384 17511 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:00:22.672533 17511 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:00:22.672621 17511 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:00:22.672693 17511 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:00:22.672753 17511 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:00:22.682325 17511 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44359
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790243735634835-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790243735634835-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:40793
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790243735634835-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40793
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790243735634835-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790243735634835-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:00:22.683673 17511 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:00:22.684929 17511 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:00:22.696192 17517 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:00:22.696256 17519 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:00:22.696234 17516 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:00:22.696386 17511 server_base.cc:1034] running on GCE node
I20260924 10:00:22.697530 17511 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:00:22.699054 17511 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:00:22.700335 17511 hybrid_clock.cc:648] HybridClock initialized: now 1790244022700262 us; error 55 us; skew 500 ppm
I20260924 10:00:22.700681 17511 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:00:22 dist-test-slave-w8d2 krb5kdc[17498](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244022, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:00:22.703675 17511 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260924 10:00:22.705061 17511 webserver.cc:460] Webserver started at http://127.0.97.62:39923/ using document root <none> and password file <none>
I20260924 10:00:22.705740 17511 fs_manager.cc:359] Metadata directory not provided
I20260924 10:00:22.705850 17511 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:00:22.706123 17511 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:00:22.709028 17511 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790243735634835-388-0/minicluster-data/master-0/data/instance:
uuid: "fd16d73ac3034b4a943cc3a6738b042c"
format_stamp: "Formatted at 2026-09-24 10:00:22 on dist-test-slave-w8d2"
server_key: "d298fc7107b3f8e36e33a9a8667e25df"
server_key_iv: "351fcb72f70422454a1fb83d6638f240"
server_key_version: "encryptionkey@0"
I20260924 10:00:22.709741 17511 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790243735634835-388-0/minicluster-data/master-0/wal/instance:
uuid: "fd16d73ac3034b4a943cc3a6738b042c"
format_stamp: "Formatted at 2026-09-24 10:00:22 on dist-test-slave-w8d2"
server_key: "d298fc7107b3f8e36e33a9a8667e25df"
server_key_iv: "351fcb72f70422454a1fb83d6638f240"
server_key_version: "encryptionkey@0"
I20260924 10:00:22.714707 17511 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260924 10:00:22.718142 17526 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:00:22.719501 17511 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 10:00:22.719655 17511 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790243735634835-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790243735634835-388-0/minicluster-data/master-0/wal
uuid: "fd16d73ac3034b4a943cc3a6738b042c"
format_stamp: "Formatted at 2026-09-24 10:00:22 on dist-test-slave-w8d2"
server_key: "d298fc7107b3f8e36e33a9a8667e25df"
server_key_iv: "351fcb72f70422454a1fb83d6638f240"
server_key_version: "encryptionkey@0"
I20260924 10:00:22.719839 17511 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790243735634835-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790243735634835-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790243735634835-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:00:22.760492 17511 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:00:22.764189 17511 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:00:22.764436 17511 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:00:22.780809 17511 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:40793
I20260924 10:00:22.780858 17577 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:40793 every 8 connection(s)
I20260924 10:00:22.782083 17511 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790243735634835-388-0/minicluster-data/master-0/data/info.pb
I20260924 10:00:22.785548 17578 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:00:22.786566 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu as pid 17511
I20260924 10:00:22.786782 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790243735634835-388-0/minicluster-data/master-0/wal/instance
I20260924 10:00:22.787629 388 external_mini_cluster.cc:1428] Setting key f8b2d65b2d99d2c9441983824c540ff5
I20260924 10:00:22.798911 17578 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd16d73ac3034b4a943cc3a6738b042c: Bootstrap starting.
Sep 24 10:00:22 dist-test-slave-w8d2 krb5kdc[17498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244022, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 10:00:22.802553 17578 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fd16d73ac3034b4a943cc3a6738b042c: Neither blocks nor log segments found. Creating new log.
I20260924 10:00:22.803570 17578 log.cc:824] T 00000000000000000000000000000000 P fd16d73ac3034b4a943cc3a6738b042c: Log is configured to *not* fsync() on all Append() calls
I20260924 10:00:22.806389 17578 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd16d73ac3034b4a943cc3a6738b042c: No bootstrap required, opened a new log
I20260924 10:00:22.809531 17581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:22.789550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58932 (local address 127.0.97.62:40793)
0924 10:00:22.790265 (+ 715us) server_negotiation.cc:206] Beginning negotiation
0924 10:00:22.790275 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 10:00:22.790316 (+ 41us) server_negotiation.cc:407] Connection header received
0924 10:00:22.791892 (+ 1576us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:00:22.791912 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:00:22.792274 (+ 362us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:00:22.792576 (+ 302us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:00:22.794199 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:00:22.795491 (+ 1292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:00:22.797259 (+ 1768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:00:22.797896 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:00:22.798012 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:00:22.802574 (+ 4562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:00:22.802581 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:00:22.802594 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:00:22.802635 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:00:22.805649 (+ 3014us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:00:22.806452 (+ 803us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:00:22.806459 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:00:22.806467 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:00:22.806623 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:00:22.807041 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:00:22.807044 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:00:22.807046 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:00:22.807541 (+ 495us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:00:22.807753 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0924 10:00:22.808220 (+ 467us) server_negotiation.cc:299] Negotiation successful
0924 10:00:22.808552 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":519,"thread_start_us":331,"threads_started":1}
I20260924 10:00:22.814543 17578 raft_consensus.cc:348] T 00000000000000000000000000000000 P fd16d73ac3034b4a943cc3a6738b042c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd16d73ac3034b4a943cc3a6738b042c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40793 } }
I20260924 10:00:22.814848 17578 raft_consensus.cc:374] T 00000000000000000000000000000000 P fd16d73ac3034b4a943cc3a6738b042c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:00:22.814936 17578 raft_consensus.cc:729] T 00000000000000000000000000000000 P fd16d73ac3034b4a943cc3a6738b042c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd16d73ac3034b4a943cc3a6738b042c, State: Initialized, Role: FOLLOWER
I20260924 10:00:22.815487 17578 consensus_queue.cc:260] T 00000000000000000000000000000000 P fd16d73ac3034b4a943cc3a6738b042c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd16d73ac3034b4a943cc3a6738b042c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40793 } }
I20260924 10:00:22.815686 17578 raft_consensus.cc:388] T 00000000000000000000000000000000 P fd16d73ac3034b4a943cc3a6738b042c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:00:22.815801 17578 raft_consensus.cc:482] T 00000000000000000000000000000000 P fd16d73ac3034b4a943cc3a6738b042c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:00:22.815937 17578 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fd16d73ac3034b4a943cc3a6738b042c [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:00:22.817898 17578 raft_consensus.cc:504] T 00000000000000000000000000000000 P fd16d73ac3034b4a943cc3a6738b042c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd16d73ac3034b4a943cc3a6738b042c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40793 } }
I20260924 10:00:22.818385 17578 leader_election.cc:302] T 00000000000000000000000000000000 P fd16d73ac3034b4a943cc3a6738b042c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fd16d73ac3034b4a943cc3a6738b042c; no voters:
I20260924 10:00:22.818895 17578 leader_election.cc:288] T 00000000000000000000000000000000 P fd16d73ac3034b4a943cc3a6738b042c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:00:22.818996 17583 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fd16d73ac3034b4a943cc3a6738b042c [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:00:22.819672 17583 raft_consensus.cc:686] T 00000000000000000000000000000000 P fd16d73ac3034b4a943cc3a6738b042c [term 1 LEADER]: Becoming Leader. State: Replica: fd16d73ac3034b4a943cc3a6738b042c, State: Running, Role: LEADER
I20260924 10:00:22.820196 17583 consensus_queue.cc:237] T 00000000000000000000000000000000 P fd16d73ac3034b4a943cc3a6738b042c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd16d73ac3034b4a943cc3a6738b042c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40793 } }
I20260924 10:00:22.820669 17578 sys_catalog.cc:564] T 00000000000000000000000000000000 P fd16d73ac3034b4a943cc3a6738b042c [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:00:22.822661 17584 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd16d73ac3034b4a943cc3a6738b042c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fd16d73ac3034b4a943cc3a6738b042c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd16d73ac3034b4a943cc3a6738b042c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40793 } } }
I20260924 10:00:22.823063 17584 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd16d73ac3034b4a943cc3a6738b042c [sys.catalog]: This master's current role is: LEADER
I20260924 10:00:22.822950 17585 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd16d73ac3034b4a943cc3a6738b042c [sys.catalog]: SysCatalogTable state changed. Reason: New leader fd16d73ac3034b4a943cc3a6738b042c. Latest consensus state: current_term: 1 leader_uuid: "fd16d73ac3034b4a943cc3a6738b042c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd16d73ac3034b4a943cc3a6738b042c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40793 } } }
I20260924 10:00:22.823382 17585 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd16d73ac3034b4a943cc3a6738b042c [sys.catalog]: This master's current role is: LEADER
I20260924 10:00:22.824182 17590 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:00:22.832973 17590 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:00:22.841038 17590 catalog_manager.cc:1347] Generated new cluster ID: 6ead5d4f91eb447e861005f4f83fb39e
I20260924 10:00:22.841141 17590 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:00:22.866168 17590 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:00:22.867511 17590 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:00:22.875952 17590 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fd16d73ac3034b4a943cc3a6738b042c: Generated new TSK 0
I20260924 10:00:22.876978 17590 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790243735634835-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790243735634835-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260924 10:00:22.929874 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
/tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790243735634835-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790243735634835-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790243735634835-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790243735634835-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40793
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790243735634835-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44359
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790243735634835-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790243735634835-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790243735634835-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:00:23.238391 17604 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:00:23.238646 17604 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:00:23.238773 17604 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:00:23.245764 17604 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:00:23.245860 17604 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:00:23.246001 17604 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260924 10:00:23.255270 17604 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44359
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790243735634835-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790243735634835-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790243735634835-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790243735634835-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40793
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790243735634835-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:00:23.256578 17604 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:00:23.257880 17604 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:00:23.269805 17610 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:00:23.269896 17609 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:00:23.269837 17612 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:00:23.270515 17604 server_base.cc:1034] running on GCE node
I20260924 10:00:23.271118 17604 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:00:23.271970 17604 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:00:23.273157 17604 hybrid_clock.cc:648] HybridClock initialized: now 1790244023273112 us; error 41 us; skew 500 ppm
I20260924 10:00:23.273427 17604 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:00:23 dist-test-slave-w8d2 krb5kdc[17498](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244023, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:00:23.276284 17604 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260924 10:00:23.277590 17604 webserver.cc:460] Webserver started at http://127.0.97.1:41789/ using document root <none> and password file <none>
I20260924 10:00:23.278214 17604 fs_manager.cc:359] Metadata directory not provided
I20260924 10:00:23.278321 17604 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:00:23.278585 17604 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:00:23.281304 17604 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790243735634835-388-0/minicluster-data/ts-0/data/instance:
uuid: "524230c8fd154a919c7a6c6b5d5cd9de"
format_stamp: "Formatted at 2026-09-24 10:00:23 on dist-test-slave-w8d2"
server_key: "81ef53b5ee4e581f467d9720c503aab7"
server_key_iv: "edb940c536a6795b8d6040250a911f51"
server_key_version: "encryptionkey@0"
I20260924 10:00:23.282007 17604 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790243735634835-388-0/minicluster-data/ts-0/wal/instance:
uuid: "524230c8fd154a919c7a6c6b5d5cd9de"
format_stamp: "Formatted at 2026-09-24 10:00:23 on dist-test-slave-w8d2"
server_key: "81ef53b5ee4e581f467d9720c503aab7"
server_key_iv: "edb940c536a6795b8d6040250a911f51"
server_key_version: "encryptionkey@0"
I20260924 10:00:23.286583 17604 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260924 10:00:23.289726 17619 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:00:23.290915 17604 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 10:00:23.291041 17604 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790243735634835-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790243735634835-388-0/minicluster-data/ts-0/wal
uuid: "524230c8fd154a919c7a6c6b5d5cd9de"
format_stamp: "Formatted at 2026-09-24 10:00:23 on dist-test-slave-w8d2"
server_key: "81ef53b5ee4e581f467d9720c503aab7"
server_key_iv: "edb940c536a6795b8d6040250a911f51"
server_key_version: "encryptionkey@0"
I20260924 10:00:23.291194 17604 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790243735634835-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790243735634835-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790243735634835-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:00:23.303406 17604 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:00:23.307286 17604 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:00:23.307512 17604 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:00:23.308362 17604 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:00:23.309679 17604 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:00:23.309748 17604 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:00:23.309844 17604 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:00:23.309890 17604 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:00:23.338759 17604 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:33261
I20260924 10:00:23.338780 17731 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:33261 every 8 connection(s)
I20260924 10:00:23.340090 17604 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790243735634835-388-0/minicluster-data/ts-0/data/info.pb
I20260924 10:00:23.350145 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu as pid 17604
I20260924 10:00:23.350337 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790243735634835-388-0/minicluster-data/ts-0/wal/instance
I20260924 10:00:23.351033 388 external_mini_cluster.cc:1428] Setting key abc5799fc46472356c57bd0aef29809d
Sep 24 10:00:23 dist-test-slave-w8d2 krb5kdc[17498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244023, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 10:00:23.359760 17734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:23.342508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:55401 (local address 127.0.97.62:40793)
0924 10:00:23.342913 (+ 405us) server_negotiation.cc:206] Beginning negotiation
0924 10:00:23.342917 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:00:23.343900 (+ 983us) server_negotiation.cc:407] Connection header received
0924 10:00:23.345711 (+ 1811us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:00:23.345717 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:00:23.345799 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:00:23.345889 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:00:23.347430 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:00:23.348171 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:00:23.349658 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:00:23.350048 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:00:23.350152 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:00:23.353723 (+ 3571us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:00:23.353730 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:00:23.353734 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:00:23.353770 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:00:23.356662 (+ 2892us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:00:23.357402 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:00:23.357408 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:00:23.357411 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:00:23.357551 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:00:23.358080 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:00:23.358085 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:00:23.358089 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:00:23.358347 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:00:23.358496 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0924 10:00:23.359172 (+ 676us) server_negotiation.cc:299] Negotiation successful
0924 10:00:23.359424 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":198,"threads_started":1}
I20260924 10:00:23.360704 17735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:23.343040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40793 (local address 127.0.97.1:55401)
0924 10:00:23.343744 (+ 704us) negotiation.cc:107] Waiting for socket to connect
0924 10:00:23.343785 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 10:00:23.345434 (+ 1649us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:00:23.346134 (+ 700us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:00:23.346143 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:00:23.346675 (+ 532us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:00:23.347266 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:00:23.347287 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:00:23.348314 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:00:23.348318 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:00:23.349526 (+ 1208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:00:23.349539 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:00:23.350258 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:00:23.350265 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:00:23.350515 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:00:23.351191 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:00:23.351212 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:00:23.353523 (+ 2311us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:00:23.356795 (+ 3272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:00:23.356802 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:00:23.356814 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:00:23.357250 (+ 436us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:00:23.357653 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:00:23.357658 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:00:23.357662 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:00:23.357956 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:00:23.358487 (+ 531us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:00:23.358494 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:00:23.358829 (+ 335us) client_negotiation.cc:769] Sending connection context
0924 10:00:23.359064 (+ 235us) client_negotiation.cc:240] Negotiation successful
0924 10:00:23.359602 (+ 538us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":291,"threads_started":1}
I20260924 10:00:23.362186 17732 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40793
I20260924 10:00:23.362504 17732 heartbeater.cc:464] Registering TS with master...
I20260924 10:00:23.363291 17732 heartbeater.cc:511] Master 127.0.97.62:40793 requested a full tablet report, sending...
I20260924 10:00:23.365458 17543 ts_manager.cc:194] Registered new tserver with Master: 524230c8fd154a919c7a6c6b5d5cd9de (127.0.97.1:33261)
I20260924 10:00:23.367358 17543 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:55401
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790243735634835-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790243735634835-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260924 10:00:23.402035 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
/tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790243735634835-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790243735634835-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790243735634835-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790243735634835-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40793
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790243735634835-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44359
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790243735634835-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790243735634835-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790243735634835-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:00:23.708088 17738 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:00:23.708334 17738 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:00:23.708451 17738 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:00:23.715492 17738 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:00:23.715584 17738 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:00:23.715749 17738 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260924 10:00:23.725103 17738 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44359
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790243735634835-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790243735634835-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790243735634835-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790243735634835-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40793
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790243735634835-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:00:23.726327 17738 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:00:23.727561 17738 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:00:23.739521 17744 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:00:23.739566 17746 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:00:23.739817 17743 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:00:23.740293 17738 server_base.cc:1034] running on GCE node
I20260924 10:00:23.740828 17738 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:00:23.741595 17738 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:00:23.742771 17738 hybrid_clock.cc:648] HybridClock initialized: now 1790244023742720 us; error 47 us; skew 500 ppm
I20260924 10:00:23.743040 17738 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:00:23 dist-test-slave-w8d2 krb5kdc[17498](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244023, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:00:23.745860 17738 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260924 10:00:23.747159 17738 webserver.cc:460] Webserver started at http://127.0.97.2:40355/ using document root <none> and password file <none>
I20260924 10:00:23.747781 17738 fs_manager.cc:359] Metadata directory not provided
I20260924 10:00:23.747879 17738 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:00:23.748157 17738 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:00:23.750788 17738 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790243735634835-388-0/minicluster-data/ts-1/data/instance:
uuid: "ee56b42aa7e44a43b557961c7a723c94"
format_stamp: "Formatted at 2026-09-24 10:00:23 on dist-test-slave-w8d2"
server_key: "3727d54a08e078c7dc452a9541f8ac38"
server_key_iv: "b20aab23b9d7b150362db36b0dea602c"
server_key_version: "encryptionkey@0"
I20260924 10:00:23.751509 17738 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790243735634835-388-0/minicluster-data/ts-1/wal/instance:
uuid: "ee56b42aa7e44a43b557961c7a723c94"
format_stamp: "Formatted at 2026-09-24 10:00:23 on dist-test-slave-w8d2"
server_key: "3727d54a08e078c7dc452a9541f8ac38"
server_key_iv: "b20aab23b9d7b150362db36b0dea602c"
server_key_version: "encryptionkey@0"
I20260924 10:00:23.756063 17738 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 10:00:23.759152 17753 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:00:23.760421 17738 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260924 10:00:23.760551 17738 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790243735634835-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790243735634835-388-0/minicluster-data/ts-1/wal
uuid: "ee56b42aa7e44a43b557961c7a723c94"
format_stamp: "Formatted at 2026-09-24 10:00:23 on dist-test-slave-w8d2"
server_key: "3727d54a08e078c7dc452a9541f8ac38"
server_key_iv: "b20aab23b9d7b150362db36b0dea602c"
server_key_version: "encryptionkey@0"
I20260924 10:00:23.760735 17738 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790243735634835-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790243735634835-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790243735634835-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:00:23.777585 17738 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:00:23.781535 17738 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:00:23.781811 17738 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:00:23.782841 17738 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:00:23.784359 17738 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:00:23.784440 17738 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:00:23.784546 17738 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:00:23.784600 17738 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:00:23.819784 17738 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:44909
I20260924 10:00:23.819826 17865 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:44909 every 8 connection(s)
I20260924 10:00:23.820955 17738 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790243735634835-388-0/minicluster-data/ts-1/data/info.pb
I20260924 10:00:23.821372 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu as pid 17738
I20260924 10:00:23.821528 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790243735634835-388-0/minicluster-data/ts-1/wal/instance
I20260924 10:00:23.821967 388 external_mini_cluster.cc:1428] Setting key 1d0dff6022ca52edf66f00bf6bd28612
Sep 24 10:00:23 dist-test-slave-w8d2 krb5kdc[17498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244023, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 10:00:23.840340 17734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:23.823695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:50347 (local address 127.0.97.62:40793)
0924 10:00:23.823893 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0924 10:00:23.823899 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:00:23.825277 (+ 1378us) server_negotiation.cc:407] Connection header received
0924 10:00:23.827149 (+ 1872us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:00:23.827155 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:00:23.827262 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:00:23.827350 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:00:23.828775 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:00:23.829500 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:00:23.831044 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:00:23.831482 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:00:23.831596 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:00:23.835041 (+ 3445us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:00:23.835051 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:00:23.835054 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:00:23.835087 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:00:23.837481 (+ 2394us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:00:23.838241 (+ 760us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:00:23.838245 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:00:23.838247 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:00:23.838377 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:00:23.838816 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:00:23.838819 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:00:23.838821 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:00:23.839060 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:00:23.839164 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0924 10:00:23.839937 (+ 773us) server_negotiation.cc:299] Negotiation successful
0924 10:00:23.840103 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260924 10:00:23.841737 17868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:23.824356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40793 (local address 127.0.97.2:50347)
0924 10:00:23.825142 (+ 786us) negotiation.cc:107] Waiting for socket to connect
0924 10:00:23.825187 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 10:00:23.826902 (+ 1715us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:00:23.827588 (+ 686us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:00:23.827596 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:00:23.828093 (+ 497us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:00:23.828617 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:00:23.828642 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:00:23.829640 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:00:23.829646 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:00:23.830899 (+ 1253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:00:23.830915 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:00:23.831710 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:00:23.831718 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:00:23.831968 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:00:23.832585 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:00:23.832613 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:00:23.834816 (+ 2203us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:00:23.837597 (+ 2781us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:00:23.837605 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:00:23.837621 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:00:23.838110 (+ 489us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:00:23.838478 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:00:23.838483 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:00:23.838486 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:00:23.838714 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:00:23.839181 (+ 467us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:00:23.839189 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:00:23.839595 (+ 406us) client_negotiation.cc:769] Sending connection context
0924 10:00:23.839883 (+ 288us) client_negotiation.cc:240] Negotiation successful
0924 10:00:23.840378 (+ 495us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":523,"thread_start_us":322,"threads_started":1}
I20260924 10:00:23.843099 17866 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40793
I20260924 10:00:23.843415 17866 heartbeater.cc:464] Registering TS with master...
I20260924 10:00:23.844098 17866 heartbeater.cc:511] Master 127.0.97.62:40793 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260924 10:00:23.845523 17543 ts_manager.cc:194] Registered new tserver with Master: ee56b42aa7e44a43b557961c7a723c94 (127.0.97.2:44909)
I20260924 10:00:23.846324 17543 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:50347
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790243735634835-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790243735634835-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260924 10:00:23.871479 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
/tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790243735634835-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790243735634835-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790243735634835-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790243735634835-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40793
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790243735634835-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44359
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790243735634835-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790243735634835-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790243735634835-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:00:24.176004 17871 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:00:24.176265 17871 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:00:24.176388 17871 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:00:24.183547 17871 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:00:24.183658 17871 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:00:24.183807 17871 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260924 10:00:24.193202 17871 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44359
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790243735634835-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790243735634835-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790243735634835-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790243735634835-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40793
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790243735634835-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:00:24.194475 17871 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:00:24.195840 17871 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:00:24.207885 17876 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:00:24.207844 17877 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:00:24.207937 17879 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:00:24.208719 17871 server_base.cc:1034] running on GCE node
I20260924 10:00:24.209205 17871 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:00:24.210012 17871 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:00:24.211192 17871 hybrid_clock.cc:648] HybridClock initialized: now 1790244024211151 us; error 31 us; skew 500 ppm
I20260924 10:00:24.211519 17871 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:00:24 dist-test-slave-w8d2 krb5kdc[17498](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244024, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:00:24.214296 17871 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260924 10:00:24.215687 17871 webserver.cc:460] Webserver started at http://127.0.97.3:33035/ using document root <none> and password file <none>
I20260924 10:00:24.216292 17871 fs_manager.cc:359] Metadata directory not provided
I20260924 10:00:24.216401 17871 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:00:24.216674 17871 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:00:24.219388 17871 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790243735634835-388-0/minicluster-data/ts-2/data/instance:
uuid: "6f64669073cd41a7ab8471bf018911ea"
format_stamp: "Formatted at 2026-09-24 10:00:24 on dist-test-slave-w8d2"
server_key: "ad58dd86ffb69e5151736812ed437e59"
server_key_iv: "6af98a169b5695337fedf0cd41924fba"
server_key_version: "encryptionkey@0"
I20260924 10:00:24.220110 17871 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790243735634835-388-0/minicluster-data/ts-2/wal/instance:
uuid: "6f64669073cd41a7ab8471bf018911ea"
format_stamp: "Formatted at 2026-09-24 10:00:24 on dist-test-slave-w8d2"
server_key: "ad58dd86ffb69e5151736812ed437e59"
server_key_iv: "6af98a169b5695337fedf0cd41924fba"
server_key_version: "encryptionkey@0"
I20260924 10:00:24.224776 17871 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 10:00:24.227880 17886 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:00:24.229072 17871 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 10:00:24.229210 17871 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790243735634835-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790243735634835-388-0/minicluster-data/ts-2/wal
uuid: "6f64669073cd41a7ab8471bf018911ea"
format_stamp: "Formatted at 2026-09-24 10:00:24 on dist-test-slave-w8d2"
server_key: "ad58dd86ffb69e5151736812ed437e59"
server_key_iv: "6af98a169b5695337fedf0cd41924fba"
server_key_version: "encryptionkey@0"
I20260924 10:00:24.229382 17871 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790243735634835-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790243735634835-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790243735634835-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:00:24.245507 17871 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:00:24.249181 17871 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:00:24.249431 17871 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:00:24.250313 17871 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:00:24.251664 17871 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:00:24.251751 17871 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:00:24.251869 17871 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:00:24.251917 17871 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:00:24.281052 17871 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:39767
I20260924 10:00:24.281096 17998 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:39767 every 8 connection(s)
I20260924 10:00:24.282212 17871 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790243735634835-388-0/minicluster-data/ts-2/data/info.pb
I20260924 10:00:24.290904 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu as pid 17871
I20260924 10:00:24.291095 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790243735634835-388-0/minicluster-data/ts-2/wal/instance
I20260924 10:00:24.291651 388 external_mini_cluster.cc:1428] Setting key 8772f7acd59cb47b7b594238c7695473
Sep 24 10:00:24 dist-test-slave-w8d2 krb5kdc[17498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244024, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 10:00:24.300976 17734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:24.284530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:51431 (local address 127.0.97.62:40793)
0924 10:00:24.284696 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0924 10:00:24.284701 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:00:24.285821 (+ 1120us) server_negotiation.cc:407] Connection header received
0924 10:00:24.287580 (+ 1759us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:00:24.287584 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:00:24.287649 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:00:24.287730 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:00:24.289125 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:00:24.289844 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:00:24.291790 (+ 1946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:00:24.292206 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:00:24.292309 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:00:24.296250 (+ 3941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:00:24.296254 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:00:24.296258 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:00:24.296289 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:00:24.298628 (+ 2339us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:00:24.299208 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:00:24.299211 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:00:24.299214 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:00:24.299344 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:00:24.299664 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:00:24.299666 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:00:24.299668 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:00:24.299917 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:00:24.300018 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0924 10:00:24.300583 (+ 565us) server_negotiation.cc:299] Negotiation successful
0924 10:00:24.300733 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 10:00:24.301846 18001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:24.284980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40793 (local address 127.0.97.3:51431)
0924 10:00:24.285660 (+ 680us) negotiation.cc:107] Waiting for socket to connect
0924 10:00:24.285702 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 10:00:24.287338 (+ 1636us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:00:24.287940 (+ 602us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:00:24.287948 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:00:24.288456 (+ 508us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:00:24.288993 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:00:24.289011 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:00:24.289974 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:00:24.289978 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:00:24.291584 (+ 1606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:00:24.291599 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:00:24.292438 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:00:24.292444 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:00:24.292694 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:00:24.293694 (+ 1000us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:00:24.293730 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:00:24.296068 (+ 2338us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:00:24.298745 (+ 2677us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:00:24.298751 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:00:24.298761 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:00:24.299122 (+ 361us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:00:24.299421 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:00:24.299424 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:00:24.299426 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:00:24.299585 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:00:24.300016 (+ 431us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:00:24.300021 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:00:24.300295 (+ 274us) client_negotiation.cc:769] Sending connection context
0924 10:00:24.300512 (+ 217us) client_negotiation.cc:240] Negotiation successful
0924 10:00:24.300843 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":279,"threads_started":1}
I20260924 10:00:24.302959 17999 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40793
I20260924 10:00:24.303208 17999 heartbeater.cc:464] Registering TS with master...
I20260924 10:00:24.303887 17999 heartbeater.cc:511] Master 127.0.97.62:40793 requested a full tablet report, sending...
I20260924 10:00:24.305181 17543 ts_manager.cc:194] Registered new tserver with Master: 6f64669073cd41a7ab8471bf018911ea (127.0.97.3:39767)
I20260924 10:00:24.306053 17543 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:51431
I20260924 10:00:24.306770 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 10:00:24.322274 17734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:24.311043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58940 (local address 127.0.97.62:40793)
0924 10:00:24.311188 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0924 10:00:24.311192 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:00:24.311659 (+ 467us) server_negotiation.cc:407] Connection header received
0924 10:00:24.311765 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:00:24.311769 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:00:24.311828 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:00:24.311906 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:00:24.313210 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:00:24.313909 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:00:24.315168 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:00:24.315601 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:00:24.315687 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:00:24.317801 (+ 2114us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:00:24.317806 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:00:24.317809 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:00:24.317838 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:00:24.320078 (+ 2240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:00:24.320707 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:00:24.320711 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:00:24.320713 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:00:24.320825 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:00:24.321106 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:00:24.321109 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:00:24.321111 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:00:24.321429 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:00:24.321610 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0924 10:00:24.321871 (+ 261us) server_negotiation.cc:299] Negotiation successful
0924 10:00:24.322024 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 10:00:24.326203 17542 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58940:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 10:00:24.332907 17542 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 10:00:24.366489 18012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:24.345089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:44909 (local address 127.0.0.1:46370)
0924 10:00:24.357813 (+ 12724us) negotiation.cc:107] Waiting for socket to connect
0924 10:00:24.357829 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 10:00:24.357947 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:00:24.358928 (+ 981us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:00:24.358932 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:00:24.358971 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:00:24.359441 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:00:24.359451 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:00:24.361318 (+ 1867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:00:24.361328 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:00:24.363626 (+ 2298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:00:24.363642 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:00:24.365539 (+ 1897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:00:24.365546 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:00:24.365723 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:00:24.365775 (+ 52us) client_negotiation.cc:769] Sending connection context
0924 10:00:24.365901 (+ 126us) client_negotiation.cc:240] Negotiation successful
0924 10:00:24.366106 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12611,"mutex_wait_us":184,"spinlock_wait_cycles":896,"thread_start_us":12483,"threads_started":1}
I20260924 10:00:24.366847 18016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:24.346194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48340 (local address 127.0.97.3:39767)
0924 10:00:24.355704 (+ 9510us) server_negotiation.cc:206] Beginning negotiation
0924 10:00:24.355712 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 10:00:24.358649 (+ 2937us) server_negotiation.cc:407] Connection header received
0924 10:00:24.358964 (+ 315us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:00:24.358972 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:00:24.359164 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:00:24.359357 (+ 193us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:00:24.360077 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:00:24.361210 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:00:24.364272 (+ 3062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:00:24.365474 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:00:24.365582 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:00:24.365799 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0924 10:00:24.366357 (+ 558us) server_negotiation.cc:299] Negotiation successful
0924 10:00:24.366535 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9305,"thread_start_us":9147,"threads_started":1}
I20260924 10:00:24.366897 18015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:24.344921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33261 (local address 127.0.0.1:54426)
0924 10:00:24.357162 (+ 12241us) negotiation.cc:107] Waiting for socket to connect
0924 10:00:24.357200 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 10:00:24.357396 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:00:24.358733 (+ 1337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:00:24.358741 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:00:24.358780 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:00:24.359370 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:00:24.359387 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:00:24.361318 (+ 1931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:00:24.361327 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:00:24.364053 (+ 2726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:00:24.364071 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:00:24.365885 (+ 1814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:00:24.365891 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:00:24.366128 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:00:24.366213 (+ 85us) client_negotiation.cc:769] Sending connection context
0924 10:00:24.366282 (+ 69us) client_negotiation.cc:240] Negotiation successful
0924 10:00:24.366454 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11987,"mutex_wait_us":244,"thread_start_us":13324,"threads_started":1}
I20260924 10:00:24.367288 18011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:24.345615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:39767 (local address 127.0.0.1:48340)
0924 10:00:24.358538 (+ 12923us) negotiation.cc:107] Waiting for socket to connect
0924 10:00:24.358556 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 10:00:24.358760 (+ 204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:00:24.359629 (+ 869us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:00:24.359632 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:00:24.359651 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:00:24.359912 (+ 261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:00:24.359919 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:00:24.361367 (+ 1448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:00:24.361372 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:00:24.364115 (+ 2743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:00:24.364125 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:00:24.365933 (+ 1808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:00:24.365939 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:00:24.366134 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:00:24.366206 (+ 72us) client_negotiation.cc:769] Sending connection context
0924 10:00:24.366292 (+ 86us) client_negotiation.cc:240] Negotiation successful
0924 10:00:24.366454 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12781,"mutex_wait_us":238,"thread_start_us":11151,"threads_started":1}
I20260924 10:00:24.367583 18014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:24.345558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46370 (local address 127.0.97.2:44909)
0924 10:00:24.355148 (+ 9590us) server_negotiation.cc:206] Beginning negotiation
0924 10:00:24.355159 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 10:00:24.358066 (+ 2907us) server_negotiation.cc:407] Connection header received
0924 10:00:24.358193 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:00:24.358202 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:00:24.358487 (+ 285us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:00:24.358690 (+ 203us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:00:24.359625 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:00:24.361166 (+ 1541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:00:24.363765 (+ 2599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:00:24.365115 (+ 1350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:00:24.365231 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:00:24.365416 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0924 10:00:24.367087 (+ 1671us) server_negotiation.cc:299] Negotiation successful
0924 10:00:24.367271 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9331,"thread_start_us":9212,"threads_started":1}
I20260924 10:00:24.367671 18013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:24.345148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54426 (local address 127.0.97.1:33261)
0924 10:00:24.358202 (+ 13054us) server_negotiation.cc:206] Beginning negotiation
0924 10:00:24.358210 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 10:00:24.358233 (+ 23us) server_negotiation.cc:407] Connection header received
0924 10:00:24.358331 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:00:24.358337 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:00:24.358501 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:00:24.359085 (+ 584us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:00:24.359540 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:00:24.361166 (+ 1626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:00:24.364204 (+ 3038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:00:24.365702 (+ 1498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:00:24.365817 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:00:24.366013 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0924 10:00:24.367203 (+ 1190us) server_negotiation.cc:299] Negotiation successful
0924 10:00:24.367372 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12893,"thread_start_us":12747,"threads_started":1}
I20260924 10:00:24.370569 17732 heartbeater.cc:503] Master 127.0.97.62:40793 was elected leader, sending a full tablet report...
I20260924 10:00:24.370654 17934 tablet_service.cc:1467] Processing CreateTablet for tablet cf4e07b120ce4c0e9dba34db69f815a5 (DEFAULT_TABLE table=test-table [id=be02c15722bc496a84bdd5e92816bba8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:00:24.370811 17801 tablet_service.cc:1467] Processing CreateTablet for tablet cf4e07b120ce4c0e9dba34db69f815a5 (DEFAULT_TABLE table=test-table [id=be02c15722bc496a84bdd5e92816bba8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:00:24.371608 17667 tablet_service.cc:1467] Processing CreateTablet for tablet cf4e07b120ce4c0e9dba34db69f815a5 (DEFAULT_TABLE table=test-table [id=be02c15722bc496a84bdd5e92816bba8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:00:24.372136 17934 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf4e07b120ce4c0e9dba34db69f815a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:00:24.372141 17801 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf4e07b120ce4c0e9dba34db69f815a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:00:24.372795 17667 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf4e07b120ce4c0e9dba34db69f815a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:00:24.383131 18017 tablet_bootstrap.cc:492] T cf4e07b120ce4c0e9dba34db69f815a5 P 6f64669073cd41a7ab8471bf018911ea: Bootstrap starting.
I20260924 10:00:24.385461 18018 tablet_bootstrap.cc:492] T cf4e07b120ce4c0e9dba34db69f815a5 P ee56b42aa7e44a43b557961c7a723c94: Bootstrap starting.
I20260924 10:00:24.386226 18019 tablet_bootstrap.cc:492] T cf4e07b120ce4c0e9dba34db69f815a5 P 524230c8fd154a919c7a6c6b5d5cd9de: Bootstrap starting.
I20260924 10:00:24.387667 18017 tablet_bootstrap.cc:654] T cf4e07b120ce4c0e9dba34db69f815a5 P 6f64669073cd41a7ab8471bf018911ea: Neither blocks nor log segments found. Creating new log.
I20260924 10:00:24.388854 18018 tablet_bootstrap.cc:654] T cf4e07b120ce4c0e9dba34db69f815a5 P ee56b42aa7e44a43b557961c7a723c94: Neither blocks nor log segments found. Creating new log.
I20260924 10:00:24.388881 18019 tablet_bootstrap.cc:654] T cf4e07b120ce4c0e9dba34db69f815a5 P 524230c8fd154a919c7a6c6b5d5cd9de: Neither blocks nor log segments found. Creating new log.
I20260924 10:00:24.388918 18017 log.cc:824] T cf4e07b120ce4c0e9dba34db69f815a5 P 6f64669073cd41a7ab8471bf018911ea: Log is configured to *not* fsync() on all Append() calls
I20260924 10:00:24.389873 18019 log.cc:824] T cf4e07b120ce4c0e9dba34db69f815a5 P 524230c8fd154a919c7a6c6b5d5cd9de: Log is configured to *not* fsync() on all Append() calls
I20260924 10:00:24.389910 18018 log.cc:824] T cf4e07b120ce4c0e9dba34db69f815a5 P ee56b42aa7e44a43b557961c7a723c94: Log is configured to *not* fsync() on all Append() calls
I20260924 10:00:24.391809 18017 tablet_bootstrap.cc:492] T cf4e07b120ce4c0e9dba34db69f815a5 P 6f64669073cd41a7ab8471bf018911ea: No bootstrap required, opened a new log
I20260924 10:00:24.392091 18017 ts_tablet_manager.cc:1397] T cf4e07b120ce4c0e9dba34db69f815a5 P 6f64669073cd41a7ab8471bf018911ea: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260924 10:00:24.392685 18018 tablet_bootstrap.cc:492] T cf4e07b120ce4c0e9dba34db69f815a5 P ee56b42aa7e44a43b557961c7a723c94: No bootstrap required, opened a new log
I20260924 10:00:24.392922 18018 ts_tablet_manager.cc:1397] T cf4e07b120ce4c0e9dba34db69f815a5 P ee56b42aa7e44a43b557961c7a723c94: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.005s
I20260924 10:00:24.398144 18019 tablet_bootstrap.cc:492] T cf4e07b120ce4c0e9dba34db69f815a5 P 524230c8fd154a919c7a6c6b5d5cd9de: No bootstrap required, opened a new log
I20260924 10:00:24.398377 18019 ts_tablet_manager.cc:1397] T cf4e07b120ce4c0e9dba34db69f815a5 P 524230c8fd154a919c7a6c6b5d5cd9de: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.010s
I20260924 10:00:24.403023 18017 raft_consensus.cc:348] T cf4e07b120ce4c0e9dba34db69f815a5 P 6f64669073cd41a7ab8471bf018911ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "524230c8fd154a919c7a6c6b5d5cd9de" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33261 } } peers { permanent_uuid: "ee56b42aa7e44a43b557961c7a723c94" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44909 } } peers { permanent_uuid: "6f64669073cd41a7ab8471bf018911ea" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39767 } }
I20260924 10:00:24.403504 18017 raft_consensus.cc:374] T cf4e07b120ce4c0e9dba34db69f815a5 P 6f64669073cd41a7ab8471bf018911ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:00:24.403604 18017 raft_consensus.cc:729] T cf4e07b120ce4c0e9dba34db69f815a5 P 6f64669073cd41a7ab8471bf018911ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f64669073cd41a7ab8471bf018911ea, State: Initialized, Role: FOLLOWER
I20260924 10:00:24.403493 18018 raft_consensus.cc:348] T cf4e07b120ce4c0e9dba34db69f815a5 P ee56b42aa7e44a43b557961c7a723c94 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "524230c8fd154a919c7a6c6b5d5cd9de" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33261 } } peers { permanent_uuid: "ee56b42aa7e44a43b557961c7a723c94" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44909 } } peers { permanent_uuid: "6f64669073cd41a7ab8471bf018911ea" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39767 } }
I20260924 10:00:24.404012 18018 raft_consensus.cc:374] T cf4e07b120ce4c0e9dba34db69f815a5 P ee56b42aa7e44a43b557961c7a723c94 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:00:24.404116 18018 raft_consensus.cc:729] T cf4e07b120ce4c0e9dba34db69f815a5 P ee56b42aa7e44a43b557961c7a723c94 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee56b42aa7e44a43b557961c7a723c94, State: Initialized, Role: FOLLOWER
I20260924 10:00:24.404214 18017 consensus_queue.cc:260] T cf4e07b120ce4c0e9dba34db69f815a5 P 6f64669073cd41a7ab8471bf018911ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "524230c8fd154a919c7a6c6b5d5cd9de" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33261 } } peers { permanent_uuid: "ee56b42aa7e44a43b557961c7a723c94" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44909 } } peers { permanent_uuid: "6f64669073cd41a7ab8471bf018911ea" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39767 } }
I20260924 10:00:24.404702 18018 consensus_queue.cc:260] T cf4e07b120ce4c0e9dba34db69f815a5 P ee56b42aa7e44a43b557961c7a723c94 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "524230c8fd154a919c7a6c6b5d5cd9de" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33261 } } peers { permanent_uuid: "ee56b42aa7e44a43b557961c7a723c94" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44909 } } peers { permanent_uuid: "6f64669073cd41a7ab8471bf018911ea" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39767 } }
I20260924 10:00:24.405326 17999 heartbeater.cc:503] Master 127.0.97.62:40793 was elected leader, sending a full tablet report...
I20260924 10:00:24.405587 18019 raft_consensus.cc:348] T cf4e07b120ce4c0e9dba34db69f815a5 P 524230c8fd154a919c7a6c6b5d5cd9de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "524230c8fd154a919c7a6c6b5d5cd9de" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33261 } } peers { permanent_uuid: "ee56b42aa7e44a43b557961c7a723c94" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44909 } } peers { permanent_uuid: "6f64669073cd41a7ab8471bf018911ea" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39767 } }
I20260924 10:00:24.405869 18019 raft_consensus.cc:374] T cf4e07b120ce4c0e9dba34db69f815a5 P 524230c8fd154a919c7a6c6b5d5cd9de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:00:24.405954 18019 raft_consensus.cc:729] T cf4e07b120ce4c0e9dba34db69f815a5 P 524230c8fd154a919c7a6c6b5d5cd9de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 524230c8fd154a919c7a6c6b5d5cd9de, State: Initialized, Role: FOLLOWER
I20260924 10:00:24.406435 18019 consensus_queue.cc:260] T cf4e07b120ce4c0e9dba34db69f815a5 P 524230c8fd154a919c7a6c6b5d5cd9de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "524230c8fd154a919c7a6c6b5d5cd9de" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33261 } } peers { permanent_uuid: "ee56b42aa7e44a43b557961c7a723c94" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44909 } } peers { permanent_uuid: "6f64669073cd41a7ab8471bf018911ea" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39767 } }
I20260924 10:00:24.408296 18019 ts_tablet_manager.cc:1428] T cf4e07b120ce4c0e9dba34db69f815a5 P 524230c8fd154a919c7a6c6b5d5cd9de: Time spent starting tablet: real 0.010s user 0.007s sys 0.004s
I20260924 10:00:24.410038 18017 ts_tablet_manager.cc:1428] T cf4e07b120ce4c0e9dba34db69f815a5 P 6f64669073cd41a7ab8471bf018911ea: Time spent starting tablet: real 0.018s user 0.014s sys 0.000s
I20260924 10:00:24.410576 17866 heartbeater.cc:503] Master 127.0.97.62:40793 was elected leader, sending a full tablet report...
I20260924 10:00:24.411412 18018 ts_tablet_manager.cc:1428] T cf4e07b120ce4c0e9dba34db69f815a5 P ee56b42aa7e44a43b557961c7a723c94: Time spent starting tablet: real 0.018s user 0.015s sys 0.003s
I20260924 10:00:24.500701 18023 raft_consensus.cc:482] T cf4e07b120ce4c0e9dba34db69f815a5 P 6f64669073cd41a7ab8471bf018911ea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 10:00:24.500991 18023 raft_consensus.cc:504] T cf4e07b120ce4c0e9dba34db69f815a5 P 6f64669073cd41a7ab8471bf018911ea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "524230c8fd154a919c7a6c6b5d5cd9de" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33261 } } peers { permanent_uuid: "ee56b42aa7e44a43b557961c7a723c94" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44909 } } peers { permanent_uuid: "6f64669073cd41a7ab8471bf018911ea" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39767 } }
I20260924 10:00:24.502596 18023 leader_election.cc:288] T cf4e07b120ce4c0e9dba34db69f815a5 P 6f64669073cd41a7ab8471bf018911ea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 524230c8fd154a919c7a6c6b5d5cd9de (127.0.97.1:33261), ee56b42aa7e44a43b557961c7a723c94 (127.0.97.2:44909)
I20260924 10:00:24.509413 18014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:24.502949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:52253 (local address 127.0.97.2:44909)
0924 10:00:24.503137 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0924 10:00:24.503143 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:00:24.503332 (+ 189us) server_negotiation.cc:407] Connection header received
0924 10:00:24.503531 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:00:24.503536 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:00:24.503613 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:00:24.503714 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:00:24.504148 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:00:24.505177 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:00:24.507465 (+ 2288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:00:24.508598 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:00:24.508730 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:00:24.508825 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0924 10:00:24.509123 (+ 298us) server_negotiation.cc:299] Negotiation successful
0924 10:00:24.509214 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 10:00:24.509521 18001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:24.502936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:44909 (local address 127.0.97.3:52253)
0924 10:00:24.503213 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0924 10:00:24.503249 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 10:00:24.503395 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:00:24.503749 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:00:24.503754 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:00:24.503791 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:00:24.504027 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:00:24.504036 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:00:24.505306 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:00:24.505310 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:00:24.507332 (+ 2022us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:00:24.507347 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:00:24.508741 (+ 1394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:00:24.508746 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:00:24.508955 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:00:24.508989 (+ 34us) client_negotiation.cc:769] Sending connection context
0924 10:00:24.509091 (+ 102us) client_negotiation.cc:240] Negotiation successful
0924 10:00:24.509226 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"mutex_wait_us":53,"spinlock_wait_cycles":21888}
I20260924 10:00:24.509773 18013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:24.502977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:37075 (local address 127.0.97.1:33261)
0924 10:00:24.503165 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0924 10:00:24.503170 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:00:24.503843 (+ 673us) server_negotiation.cc:407] Connection header received
0924 10:00:24.504002 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:00:24.504006 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:00:24.504062 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:00:24.504139 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:00:24.504564 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:00:24.505473 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:00:24.507704 (+ 2231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:00:24.508871 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:00:24.508974 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:00:24.509043 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 10:00:24.509432 (+ 389us) server_negotiation.cc:299] Negotiation successful
0924 10:00:24.509554 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 10:00:24.509868 18026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:24.503032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33261 (local address 127.0.97.3:37075)
0924 10:00:24.503740 (+ 708us) negotiation.cc:107] Waiting for socket to connect
0924 10:00:24.503755 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 10:00:24.503880 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:00:24.504167 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:00:24.504170 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:00:24.504189 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:00:24.504429 (+ 240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:00:24.504436 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:00:24.505619 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:00:24.505625 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:00:24.507577 (+ 1952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:00:24.507587 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:00:24.509096 (+ 1509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:00:24.509104 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:00:24.509270 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:00:24.509299 (+ 29us) client_negotiation.cc:769] Sending connection context
0924 10:00:24.509386 (+ 87us) client_negotiation.cc:240] Negotiation successful
0924 10:00:24.509499 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":602,"mutex_wait_us":22,"thread_start_us":400,"threads_started":1}
I20260924 10:00:24.510425 17821 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf4e07b120ce4c0e9dba34db69f815a5" candidate_uuid: "6f64669073cd41a7ab8471bf018911ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee56b42aa7e44a43b557961c7a723c94" is_pre_election: true
I20260924 10:00:24.510665 17687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf4e07b120ce4c0e9dba34db69f815a5" candidate_uuid: "6f64669073cd41a7ab8471bf018911ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "524230c8fd154a919c7a6c6b5d5cd9de" is_pre_election: true
I20260924 10:00:24.510871 17821 raft_consensus.cc:2454] T cf4e07b120ce4c0e9dba34db69f815a5 P ee56b42aa7e44a43b557961c7a723c94 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6f64669073cd41a7ab8471bf018911ea in term 0.
I20260924 10:00:24.511147 17687 raft_consensus.cc:2454] T cf4e07b120ce4c0e9dba34db69f815a5 P 524230c8fd154a919c7a6c6b5d5cd9de [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6f64669073cd41a7ab8471bf018911ea in term 0.
I20260924 10:00:24.511512 17890 leader_election.cc:302] T cf4e07b120ce4c0e9dba34db69f815a5 P 6f64669073cd41a7ab8471bf018911ea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6f64669073cd41a7ab8471bf018911ea, ee56b42aa7e44a43b557961c7a723c94; no voters:
I20260924 10:00:24.511861 18023 raft_consensus.cc:2781] T cf4e07b120ce4c0e9dba34db69f815a5 P 6f64669073cd41a7ab8471bf018911ea [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 10:00:24.512015 18023 raft_consensus.cc:482] T cf4e07b120ce4c0e9dba34db69f815a5 P 6f64669073cd41a7ab8471bf018911ea [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 10:00:24.512104 18023 raft_consensus.cc:3037] T cf4e07b120ce4c0e9dba34db69f815a5 P 6f64669073cd41a7ab8471bf018911ea [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:00:24.514158 18023 raft_consensus.cc:504] T cf4e07b120ce4c0e9dba34db69f815a5 P 6f64669073cd41a7ab8471bf018911ea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "524230c8fd154a919c7a6c6b5d5cd9de" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33261 } } peers { permanent_uuid: "ee56b42aa7e44a43b557961c7a723c94" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44909 } } peers { permanent_uuid: "6f64669073cd41a7ab8471bf018911ea" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39767 } }
I20260924 10:00:24.514844 18023 leader_election.cc:288] T cf4e07b120ce4c0e9dba34db69f815a5 P 6f64669073cd41a7ab8471bf018911ea [CANDIDATE]: Term 1 election: Requested vote from peers 524230c8fd154a919c7a6c6b5d5cd9de (127.0.97.1:33261), ee56b42aa7e44a43b557961c7a723c94 (127.0.97.2:44909)
I20260924 10:00:24.515271 17821 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf4e07b120ce4c0e9dba34db69f815a5" candidate_uuid: "6f64669073cd41a7ab8471bf018911ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee56b42aa7e44a43b557961c7a723c94"
I20260924 10:00:24.515280 17687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf4e07b120ce4c0e9dba34db69f815a5" candidate_uuid: "6f64669073cd41a7ab8471bf018911ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "524230c8fd154a919c7a6c6b5d5cd9de"
I20260924 10:00:24.515472 17687 raft_consensus.cc:3037] T cf4e07b120ce4c0e9dba34db69f815a5 P 524230c8fd154a919c7a6c6b5d5cd9de [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:00:24.515472 17821 raft_consensus.cc:3037] T cf4e07b120ce4c0e9dba34db69f815a5 P ee56b42aa7e44a43b557961c7a723c94 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:00:24.518515 17821 raft_consensus.cc:2454] T cf4e07b120ce4c0e9dba34db69f815a5 P ee56b42aa7e44a43b557961c7a723c94 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6f64669073cd41a7ab8471bf018911ea in term 1.
I20260924 10:00:24.518512 17687 raft_consensus.cc:2454] T cf4e07b120ce4c0e9dba34db69f815a5 P 524230c8fd154a919c7a6c6b5d5cd9de [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6f64669073cd41a7ab8471bf018911ea in term 1.
I20260924 10:00:24.518847 17890 leader_election.cc:302] T cf4e07b120ce4c0e9dba34db69f815a5 P 6f64669073cd41a7ab8471bf018911ea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6f64669073cd41a7ab8471bf018911ea, ee56b42aa7e44a43b557961c7a723c94; no voters:
I20260924 10:00:24.519119 18023 raft_consensus.cc:2781] T cf4e07b120ce4c0e9dba34db69f815a5 P 6f64669073cd41a7ab8471bf018911ea [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:00:24.520045 18023 raft_consensus.cc:686] T cf4e07b120ce4c0e9dba34db69f815a5 P 6f64669073cd41a7ab8471bf018911ea [term 1 LEADER]: Becoming Leader. State: Replica: 6f64669073cd41a7ab8471bf018911ea, State: Running, Role: LEADER
I20260924 10:00:24.520569 18023 consensus_queue.cc:237] T cf4e07b120ce4c0e9dba34db69f815a5 P 6f64669073cd41a7ab8471bf018911ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "524230c8fd154a919c7a6c6b5d5cd9de" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33261 } } peers { permanent_uuid: "ee56b42aa7e44a43b557961c7a723c94" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44909 } } peers { permanent_uuid: "6f64669073cd41a7ab8471bf018911ea" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39767 } }
I20260924 10:00:24.526464 17542 catalog_manager.cc:5495] T cf4e07b120ce4c0e9dba34db69f815a5 P 6f64669073cd41a7ab8471bf018911ea reported cstate change: term changed from 0 to 1, leader changed from <none> to 6f64669073cd41a7ab8471bf018911ea (127.0.97.3). New cstate: current_term: 1 leader_uuid: "6f64669073cd41a7ab8471bf018911ea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "524230c8fd154a919c7a6c6b5d5cd9de" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee56b42aa7e44a43b557961c7a723c94" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44909 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6f64669073cd41a7ab8471bf018911ea" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39767 } health_report { overall_health: HEALTHY } } }
W20260924 10:00:24.533620 18000 tablet.cc:2369] T cf4e07b120ce4c0e9dba34db69f815a5 P 6f64669073cd41a7ab8471bf018911ea: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 10:00:24.565210 18016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:24.560060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48342 (local address 127.0.97.3:39767)
0924 10:00:24.560273 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0924 10:00:24.560278 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:00:24.560295 (+ 17us) server_negotiation.cc:407] Connection header received
0924 10:00:24.560488 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:00:24.560492 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:00:24.560576 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:00:24.560696 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:00:24.561087 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:00:24.561788 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:00:24.563215 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:00:24.563653 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:00:24.563746 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:00:24.564189 (+ 443us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:00:24.564694 (+ 505us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:00:24.564796 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0924 10:00:24.564865 (+ 69us) server_negotiation.cc:299] Negotiation successful
0924 10:00:24.564940 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260924 10:00:24.572714 17687 raft_consensus.cc:1261] T cf4e07b120ce4c0e9dba34db69f815a5 P 524230c8fd154a919c7a6c6b5d5cd9de [term 1 FOLLOWER]: Refusing update from remote peer 6f64669073cd41a7ab8471bf018911ea: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:00:24.572726 17821 raft_consensus.cc:1261] T cf4e07b120ce4c0e9dba34db69f815a5 P ee56b42aa7e44a43b557961c7a723c94 [term 1 FOLLOWER]: Refusing update from remote peer 6f64669073cd41a7ab8471bf018911ea: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260924 10:00:24.573642 17867 tablet.cc:2369] T cf4e07b120ce4c0e9dba34db69f815a5 P ee56b42aa7e44a43b557961c7a723c94: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 10:00:24.573531 18027 consensus_queue.cc:1035] T cf4e07b120ce4c0e9dba34db69f815a5 P 6f64669073cd41a7ab8471bf018911ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee56b42aa7e44a43b557961c7a723c94" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:00:24.573953 18023 consensus_queue.cc:1035] T cf4e07b120ce4c0e9dba34db69f815a5 P 6f64669073cd41a7ab8471bf018911ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "524230c8fd154a919c7a6c6b5d5cd9de" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:00:24.585011 18030 mvcc.cc:205] Tried to move back new op lower bound from 7332839524641763328 to 7332839524442378240. Current Snapshot: MvccSnapshot[applied={T|T < 7332839524641763328}]
I20260924 10:00:24.596407 17543 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58940:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:00:24.596632 17543 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58940:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:00:24.599855 17543 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fd16d73ac3034b4a943cc3a6738b042c: Sending DeleteTablet for 3 replicas of tablet cf4e07b120ce4c0e9dba34db69f815a5
I20260924 10:00:24.600792 17667 tablet_service.cc:1514] Processing DeleteTablet for tablet cf4e07b120ce4c0e9dba34db69f815a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:00:24 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:54426
I20260924 10:00:24.600925 17801 tablet_service.cc:1514] Processing DeleteTablet for tablet cf4e07b120ce4c0e9dba34db69f815a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:00:24 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:46370
I20260924 10:00:24.601058 17934 tablet_service.cc:1514] Processing DeleteTablet for tablet cf4e07b120ce4c0e9dba34db69f815a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:00:24 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:48340
I20260924 10:00:24.603638 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu pid 17604
I20260924 10:00:24.616291 18041 tablet_replica.cc:330] T cf4e07b120ce4c0e9dba34db69f815a5 P ee56b42aa7e44a43b557961c7a723c94: stopping tablet replica
I20260924 10:00:24.616722 18041 raft_consensus.cc:2229] T cf4e07b120ce4c0e9dba34db69f815a5 P ee56b42aa7e44a43b557961c7a723c94 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:00:24.617233 18041 raft_consensus.cc:2258] T cf4e07b120ce4c0e9dba34db69f815a5 P ee56b42aa7e44a43b557961c7a723c94 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:00:24.619374 18041 ts_tablet_manager.cc:1905] T cf4e07b120ce4c0e9dba34db69f815a5 P ee56b42aa7e44a43b557961c7a723c94: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:00:24.626191 18041 ts_tablet_manager.cc:1918] T cf4e07b120ce4c0e9dba34db69f815a5 P ee56b42aa7e44a43b557961c7a723c94: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:00:24.626355 18041 log.cc:1196] T cf4e07b120ce4c0e9dba34db69f815a5 P ee56b42aa7e44a43b557961c7a723c94: Deleting WAL directory at /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790243735634835-388-0/minicluster-data/ts-1/wal/wals/cf4e07b120ce4c0e9dba34db69f815a5
I20260924 10:00:24.626845 18041 ts_tablet_manager.cc:1939] T cf4e07b120ce4c0e9dba34db69f815a5 P ee56b42aa7e44a43b557961c7a723c94: Deleting consensus metadata
I20260924 10:00:24.628592 17530 catalog_manager.cc:4841] TS ee56b42aa7e44a43b557961c7a723c94 (127.0.97.2:44909): tablet cf4e07b120ce4c0e9dba34db69f815a5 (table test-table [id=be02c15722bc496a84bdd5e92816bba8]) successfully deleted
I20260924 10:00:24.629823 18040 tablet_replica.cc:330] T cf4e07b120ce4c0e9dba34db69f815a5 P 524230c8fd154a919c7a6c6b5d5cd9de: stopping tablet replica
I20260924 10:00:24.630386 18040 raft_consensus.cc:2229] T cf4e07b120ce4c0e9dba34db69f815a5 P 524230c8fd154a919c7a6c6b5d5cd9de [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:00:24.631038 18040 raft_consensus.cc:2258] T cf4e07b120ce4c0e9dba34db69f815a5 P 524230c8fd154a919c7a6c6b5d5cd9de [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:00:24.633850 18040 ts_tablet_manager.cc:1905] T cf4e07b120ce4c0e9dba34db69f815a5 P 524230c8fd154a919c7a6c6b5d5cd9de: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:00:24.637032 18042 tablet_replica.cc:330] T cf4e07b120ce4c0e9dba34db69f815a5 P 6f64669073cd41a7ab8471bf018911ea: stopping tablet replica
I20260924 10:00:24.637486 18042 raft_consensus.cc:2229] T cf4e07b120ce4c0e9dba34db69f815a5 P 6f64669073cd41a7ab8471bf018911ea [term 1 LEADER]: Raft consensus shutting down.
I20260924 10:00:24.638216 18042 raft_consensus.cc:2258] T cf4e07b120ce4c0e9dba34db69f815a5 P 6f64669073cd41a7ab8471bf018911ea [term 1 FOLLOWER]: Raft consensus is shut down!
Sep 24 10:00:24 dist-test-slave-w8d2 krb5kdc[17498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244022, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260924 10:00:24.640292 18042 ts_tablet_manager.cc:1905] T cf4e07b120ce4c0e9dba34db69f815a5 P 6f64669073cd41a7ab8471bf018911ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:00:24.641330 18040 ts_tablet_manager.cc:1918] T cf4e07b120ce4c0e9dba34db69f815a5 P 524230c8fd154a919c7a6c6b5d5cd9de: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:00:24.641508 18040 log.cc:1196] T cf4e07b120ce4c0e9dba34db69f815a5 P 524230c8fd154a919c7a6c6b5d5cd9de: Deleting WAL directory at /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790243735634835-388-0/minicluster-data/ts-0/wal/wals/cf4e07b120ce4c0e9dba34db69f815a5
I20260924 10:00:24.641992 18040 ts_tablet_manager.cc:1939] T cf4e07b120ce4c0e9dba34db69f815a5 P 524230c8fd154a919c7a6c6b5d5cd9de: Deleting consensus metadata
I20260924 10:00:24.643783 17527 catalog_manager.cc:4841] TS 524230c8fd154a919c7a6c6b5d5cd9de (127.0.97.1:33261): tablet cf4e07b120ce4c0e9dba34db69f815a5 (table test-table [id=be02c15722bc496a84bdd5e92816bba8]) successfully deleted
I20260924 10:00:24.646745 18042 ts_tablet_manager.cc:1918] T cf4e07b120ce4c0e9dba34db69f815a5 P 6f64669073cd41a7ab8471bf018911ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:00:24.646942 18042 log.cc:1196] T cf4e07b120ce4c0e9dba34db69f815a5 P 6f64669073cd41a7ab8471bf018911ea: Deleting WAL directory at /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790243735634835-388-0/minicluster-data/ts-2/wal/wals/cf4e07b120ce4c0e9dba34db69f815a5
I20260924 10:00:24.647523 18042 ts_tablet_manager.cc:1939] T cf4e07b120ce4c0e9dba34db69f815a5 P 6f64669073cd41a7ab8471bf018911ea: Deleting consensus metadata
I20260924 10:00:24.647935 18013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:24.605088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54432 (local address 127.0.97.1:33261)
0924 10:00:24.605274 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0924 10:00:24.605280 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:00:24.628755 (+ 23475us) server_negotiation.cc:407] Connection header received
0924 10:00:24.629365 (+ 610us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:00:24.629371 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:00:24.629464 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:00:24.629579 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:00:24.631347 (+ 1768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:00:24.632597 (+ 1250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:00:24.634707 (+ 2110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:00:24.635400 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:00:24.635497 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:00:24.641029 (+ 5532us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:00:24.641044 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:00:24.641062 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:00:24.641104 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:00:24.644933 (+ 3829us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:00:24.645723 (+ 790us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:00:24.645732 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:00:24.645735 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:00:24.645900 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:00:24.646296 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:00:24.646301 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:00:24.646304 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:00:24.646728 (+ 424us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:00:24.647014 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0924 10:00:24.647363 (+ 349us) server_negotiation.cc:299] Negotiation successful
0924 10:00:24.647572 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 10:00:24.648614 17727 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:00:24.649353 17528 catalog_manager.cc:4841] TS 6f64669073cd41a7ab8471bf018911ea (127.0.97.3:39767): tablet cf4e07b120ce4c0e9dba34db69f815a5 (table test-table [id=be02c15722bc496a84bdd5e92816bba8]) successfully deleted
I20260924 10:00:24.798861 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu with pid 17604
I20260924 10:00:24.813153 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu pid 17738
Sep 24 10:00:24 dist-test-slave-w8d2 krb5kdc[17498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244022, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260924 10:00:24.830026 18014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:24.813869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46372 (local address 127.0.97.2:44909)
0924 10:00:24.814051 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0924 10:00:24.814057 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:00:24.814125 (+ 68us) server_negotiation.cc:407] Connection header received
0924 10:00:24.814318 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:00:24.814322 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:00:24.814388 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:00:24.814469 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:00:24.815744 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:00:24.816491 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:00:24.817717 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:00:24.818119 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:00:24.818234 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:00:24.823392 (+ 5158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:00:24.823405 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:00:24.823421 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:00:24.823463 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:00:24.827156 (+ 3693us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:00:24.827970 (+ 814us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:00:24.827979 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:00:24.827983 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:00:24.828110 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:00:24.828460 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:00:24.828465 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:00:24.828469 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:00:24.828919 (+ 450us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:00:24.829178 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0924 10:00:24.829424 (+ 246us) server_negotiation.cc:299] Negotiation successful
0924 10:00:24.829648 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260924 10:00:24.830654 17861 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:00:24.919024 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu with pid 17738
I20260924 10:00:24.933015 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu pid 17871
Sep 24 10:00:24 dist-test-slave-w8d2 krb5kdc[17498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244022, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260924 10:00:24.950687 18016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:24.933847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48346 (local address 127.0.97.3:39767)
0924 10:00:24.934001 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0924 10:00:24.934006 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:00:24.934104 (+ 98us) server_negotiation.cc:407] Connection header received
0924 10:00:24.934319 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:00:24.934323 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:00:24.934392 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:00:24.934473 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:00:24.936197 (+ 1724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:00:24.937001 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:00:24.938845 (+ 1844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:00:24.939422 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:00:24.939521 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:00:24.944779 (+ 5258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:00:24.944792 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:00:24.944808 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:00:24.944845 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:00:24.947765 (+ 2920us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:00:24.948591 (+ 826us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:00:24.948599 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:00:24.948601 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:00:24.948768 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:00:24.949213 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:00:24.949217 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:00:24.949219 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:00:24.949578 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:00:24.949835 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0924 10:00:24.950177 (+ 342us) server_negotiation.cc:299] Negotiation successful
0924 10:00:24.950359 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 10:00:24.951532 17994 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:00:25.072441 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu with pid 17871
I20260924 10:00:25.087739 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu pid 17511
I20260924 10:00:25.088762 17573 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:00:25.195734 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu with pid 17511
2026-09-24T10:00:25Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (4049 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-24T10:00:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:00:25Z Disabled control of system clock
I20260924 10:00:25.261739 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
/tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790243735634835-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790243735634835-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790243735634835-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790243735634835-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33429
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:44485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:33429
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 10:00:25.569595 18051 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:00:25.569893 18051 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:00:25.569996 18051 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:00:25.577078 18051 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:00:25.577198 18051 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:00:25.577258 18051 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:00:25.577346 18051 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:00:25.577417 18051 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:00:25.586468 18051 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790243735634835-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790243735634835-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:33429
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33429
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790243735634835-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790243735634835-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:00:25.587751 18051 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:00:25.589452 18051 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:00:25.600317 18057 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:00:25.600347 18059 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:00:25.600740 18056 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:00:25.601109 18051 server_base.cc:1034] running on GCE node
I20260924 10:00:25.601809 18051 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:00:25.603111 18051 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:00:25.604328 18051 hybrid_clock.cc:648] HybridClock initialized: now 1790244025604274 us; error 57 us; skew 500 ppm
I20260924 10:00:25.604636 18051 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 10:00:25.606659 18051 webserver.cc:460] Webserver started at http://127.0.97.62:46451/ using document root <none> and password file <none>
I20260924 10:00:25.607317 18051 fs_manager.cc:359] Metadata directory not provided
I20260924 10:00:25.607417 18051 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:00:25.607697 18051 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:00:25.610461 18051 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790243735634835-388-0/minicluster-data/master-0/data/instance:
uuid: "929d96eb44f74ae2b1ffca93d2f5eb4f"
format_stamp: "Formatted at 2026-09-24 10:00:25 on dist-test-slave-w8d2"
server_key: "8502a41332d4f614ce37808f37f19332"
server_key_iv: "825fbb258f645293ecc708bd2800eb1e"
server_key_version: "encryptionkey@0"
I20260924 10:00:25.611193 18051 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790243735634835-388-0/minicluster-data/master-0/wal/instance:
uuid: "929d96eb44f74ae2b1ffca93d2f5eb4f"
format_stamp: "Formatted at 2026-09-24 10:00:25 on dist-test-slave-w8d2"
server_key: "8502a41332d4f614ce37808f37f19332"
server_key_iv: "825fbb258f645293ecc708bd2800eb1e"
server_key_version: "encryptionkey@0"
I20260924 10:00:25.615875 18051 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260924 10:00:25.618979 18065 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:00:25.620208 18051 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260924 10:00:25.620373 18051 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790243735634835-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790243735634835-388-0/minicluster-data/master-0/wal
uuid: "929d96eb44f74ae2b1ffca93d2f5eb4f"
format_stamp: "Formatted at 2026-09-24 10:00:25 on dist-test-slave-w8d2"
server_key: "8502a41332d4f614ce37808f37f19332"
server_key_iv: "825fbb258f645293ecc708bd2800eb1e"
server_key_version: "encryptionkey@0"
I20260924 10:00:25.620548 18051 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790243735634835-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790243735634835-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790243735634835-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:00:25.640904 18051 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:00:25.641700 18051 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:00:25.641984 18051 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:00:25.658195 18051 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:33429
I20260924 10:00:25.658212 18116 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:33429 every 8 connection(s)
I20260924 10:00:25.659457 18051 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790243735634835-388-0/minicluster-data/master-0/data/info.pb
I20260924 10:00:25.660840 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu as pid 18051
I20260924 10:00:25.660992 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790243735634835-388-0/minicluster-data/master-0/wal/instance
I20260924 10:00:25.661386 388 external_mini_cluster.cc:1428] Setting key af288e3918fedc3ee41daaa51ddbb918
I20260924 10:00:25.663334 18117 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:00:25.674372 18120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:25.663131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43790 (local address 127.0.97.62:33429)
0924 10:00:25.664078 (+ 947us) server_negotiation.cc:206] Beginning negotiation
0924 10:00:25.664089 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 10:00:25.664127 (+ 38us) server_negotiation.cc:407] Connection header received
0924 10:00:25.665657 (+ 1530us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:00:25.665677 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:00:25.666099 (+ 422us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:00:25.666431 (+ 332us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:00:25.666949 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:00:25.668288 (+ 1339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:00:25.670240 (+ 1952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:00:25.670867 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:00:25.670993 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:00:25.672182 (+ 1189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:00:25.672189 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:00:25.672202 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:00:25.672752 (+ 550us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:00:25.672829 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:00:25.672850 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:00:25.672929 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0924 10:00:25.673152 (+ 223us) server_negotiation.cc:299] Negotiation successful
0924 10:00:25.673471 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":561,"thread_start_us":344,"threads_started":1}
I20260924 10:00:25.675535 18117 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 929d96eb44f74ae2b1ffca93d2f5eb4f: Bootstrap starting.
I20260924 10:00:25.678617 18117 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 929d96eb44f74ae2b1ffca93d2f5eb4f: Neither blocks nor log segments found. Creating new log.
I20260924 10:00:25.679551 18117 log.cc:824] T 00000000000000000000000000000000 P 929d96eb44f74ae2b1ffca93d2f5eb4f: Log is configured to *not* fsync() on all Append() calls
I20260924 10:00:25.682128 18117 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 929d96eb44f74ae2b1ffca93d2f5eb4f: No bootstrap required, opened a new log
I20260924 10:00:25.689137 18117 raft_consensus.cc:348] T 00000000000000000000000000000000 P 929d96eb44f74ae2b1ffca93d2f5eb4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "929d96eb44f74ae2b1ffca93d2f5eb4f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33429 } }
I20260924 10:00:25.689512 18117 raft_consensus.cc:374] T 00000000000000000000000000000000 P 929d96eb44f74ae2b1ffca93d2f5eb4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:00:25.689599 18117 raft_consensus.cc:729] T 00000000000000000000000000000000 P 929d96eb44f74ae2b1ffca93d2f5eb4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 929d96eb44f74ae2b1ffca93d2f5eb4f, State: Initialized, Role: FOLLOWER
I20260924 10:00:25.690176 18117 consensus_queue.cc:260] T 00000000000000000000000000000000 P 929d96eb44f74ae2b1ffca93d2f5eb4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "929d96eb44f74ae2b1ffca93d2f5eb4f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33429 } }
I20260924 10:00:25.690392 18117 raft_consensus.cc:388] T 00000000000000000000000000000000 P 929d96eb44f74ae2b1ffca93d2f5eb4f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:00:25.690517 18117 raft_consensus.cc:482] T 00000000000000000000000000000000 P 929d96eb44f74ae2b1ffca93d2f5eb4f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:00:25.690657 18117 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 929d96eb44f74ae2b1ffca93d2f5eb4f [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:00:25.692792 18117 raft_consensus.cc:504] T 00000000000000000000000000000000 P 929d96eb44f74ae2b1ffca93d2f5eb4f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "929d96eb44f74ae2b1ffca93d2f5eb4f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33429 } }
I20260924 10:00:25.693249 18117 leader_election.cc:302] T 00000000000000000000000000000000 P 929d96eb44f74ae2b1ffca93d2f5eb4f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 929d96eb44f74ae2b1ffca93d2f5eb4f; no voters:
I20260924 10:00:25.693778 18117 leader_election.cc:288] T 00000000000000000000000000000000 P 929d96eb44f74ae2b1ffca93d2f5eb4f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:00:25.693917 18122 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 929d96eb44f74ae2b1ffca93d2f5eb4f [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:00:25.694536 18122 raft_consensus.cc:686] T 00000000000000000000000000000000 P 929d96eb44f74ae2b1ffca93d2f5eb4f [term 1 LEADER]: Becoming Leader. State: Replica: 929d96eb44f74ae2b1ffca93d2f5eb4f, State: Running, Role: LEADER
I20260924 10:00:25.695053 18122 consensus_queue.cc:237] T 00000000000000000000000000000000 P 929d96eb44f74ae2b1ffca93d2f5eb4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "929d96eb44f74ae2b1ffca93d2f5eb4f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33429 } }
I20260924 10:00:25.695984 18117 sys_catalog.cc:564] T 00000000000000000000000000000000 P 929d96eb44f74ae2b1ffca93d2f5eb4f [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:00:25.697782 18124 sys_catalog.cc:455] T 00000000000000000000000000000000 P 929d96eb44f74ae2b1ffca93d2f5eb4f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 929d96eb44f74ae2b1ffca93d2f5eb4f. Latest consensus state: current_term: 1 leader_uuid: "929d96eb44f74ae2b1ffca93d2f5eb4f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "929d96eb44f74ae2b1ffca93d2f5eb4f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33429 } } }
I20260924 10:00:25.698189 18124 sys_catalog.cc:458] T 00000000000000000000000000000000 P 929d96eb44f74ae2b1ffca93d2f5eb4f [sys.catalog]: This master's current role is: LEADER
I20260924 10:00:25.698591 18123 sys_catalog.cc:455] T 00000000000000000000000000000000 P 929d96eb44f74ae2b1ffca93d2f5eb4f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "929d96eb44f74ae2b1ffca93d2f5eb4f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "929d96eb44f74ae2b1ffca93d2f5eb4f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33429 } } }
I20260924 10:00:25.698865 18123 sys_catalog.cc:458] T 00000000000000000000000000000000 P 929d96eb44f74ae2b1ffca93d2f5eb4f [sys.catalog]: This master's current role is: LEADER
I20260924 10:00:25.700217 18129 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:00:25.704795 18129 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:00:25.716550 18129 catalog_manager.cc:1347] Generated new cluster ID: ea49221c47cb4610860240410a6c2f04
I20260924 10:00:25.716678 18129 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:00:25.729149 18129 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:00:25.730881 18129 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:00:25.740768 18129 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 929d96eb44f74ae2b1ffca93d2f5eb4f: Generated new TSK 0
I20260924 10:00:25.741699 18129 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 10:00:25.752233 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
/tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790243735634835-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790243735634835-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790243735634835-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790243735634835-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33429
--builtin_ntp_servers=127.0.97.20:44485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 10:00:26.063140 18141 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:00:26.063431 18141 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:00:26.063566 18141 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:00:26.070588 18141 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:00:26.070688 18141 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:00:26.070834 18141 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260924 10:00:26.080122 18141 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790243735634835-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790243735634835-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790243735634835-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33429
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790243735634835-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:00:26.081394 18141 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:00:26.083082 18141 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:00:26.095384 18149 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:00:26.095439 18146 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:00:26.095414 18147 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:00:26.096051 18141 server_base.cc:1034] running on GCE node
I20260924 10:00:26.096642 18141 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:00:26.097568 18141 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:00:26.098768 18141 hybrid_clock.cc:648] HybridClock initialized: now 1790244026098707 us; error 50 us; skew 500 ppm
I20260924 10:00:26.099078 18141 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 10:00:26.104367 18141 webserver.cc:460] Webserver started at http://127.0.97.1:46863/ using document root <none> and password file <none>
I20260924 10:00:26.104996 18141 fs_manager.cc:359] Metadata directory not provided
I20260924 10:00:26.105103 18141 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:00:26.105396 18141 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:00:26.108286 18141 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790243735634835-388-0/minicluster-data/ts-0/data/instance:
uuid: "23eab3890a774185a370ebe62fe5f762"
format_stamp: "Formatted at 2026-09-24 10:00:26 on dist-test-slave-w8d2"
server_key: "074173a95b903546ae181503776dd0c1"
server_key_iv: "b016f0cc136a15130fb5c7cac2b1c957"
server_key_version: "encryptionkey@0"
I20260924 10:00:26.109012 18141 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790243735634835-388-0/minicluster-data/ts-0/wal/instance:
uuid: "23eab3890a774185a370ebe62fe5f762"
format_stamp: "Formatted at 2026-09-24 10:00:26 on dist-test-slave-w8d2"
server_key: "074173a95b903546ae181503776dd0c1"
server_key_iv: "b016f0cc136a15130fb5c7cac2b1c957"
server_key_version: "encryptionkey@0"
I20260924 10:00:26.113991 18141 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260924 10:00:26.117408 18155 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:00:26.118752 18141 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260924 10:00:26.118896 18141 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790243735634835-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790243735634835-388-0/minicluster-data/ts-0/wal
uuid: "23eab3890a774185a370ebe62fe5f762"
format_stamp: "Formatted at 2026-09-24 10:00:26 on dist-test-slave-w8d2"
server_key: "074173a95b903546ae181503776dd0c1"
server_key_iv: "b016f0cc136a15130fb5c7cac2b1c957"
server_key_version: "encryptionkey@0"
I20260924 10:00:26.119102 18141 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790243735634835-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790243735634835-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790243735634835-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:00:26.135468 18141 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:00:26.136219 18141 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:00:26.136458 18141 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:00:26.137373 18141 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:00:26.138734 18141 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:00:26.138828 18141 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:00:26.138926 18141 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:00:26.138996 18141 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:00:26.166476 18141 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:46719
I20260924 10:00:26.166519 18267 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:46719 every 8 connection(s)
I20260924 10:00:26.167723 18141 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790243735634835-388-0/minicluster-data/ts-0/data/info.pb
I20260924 10:00:26.173651 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu as pid 18141
I20260924 10:00:26.173812 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790243735634835-388-0/minicluster-data/ts-0/wal/instance
I20260924 10:00:26.174365 388 external_mini_cluster.cc:1428] Setting key 2d6b598371ba1f6c84323f295d47faeb
I20260924 10:00:26.177214 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
/tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790243735634835-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790243735634835-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790243735634835-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790243735634835-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33429
--builtin_ntp_servers=127.0.97.20:44485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 10:00:26.180154 18120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:26.170161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:46449 (local address 127.0.97.62:33429)
0924 10:00:26.170325 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0924 10:00:26.170330 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:00:26.171618 (+ 1288us) server_negotiation.cc:407] Connection header received
0924 10:00:26.173330 (+ 1712us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:00:26.173335 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:00:26.173413 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:00:26.173498 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:00:26.174576 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:00:26.175428 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:00:26.177639 (+ 2211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:00:26.178035 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:00:26.178130 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:00:26.179530 (+ 1400us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:00:26.179534 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:00:26.179537 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:00:26.179554 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:00:26.179584 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:00:26.179606 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:00:26.179671 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0924 10:00:26.179867 (+ 196us) server_negotiation.cc:299] Negotiation successful
0924 10:00:26.179947 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 10:00:26.181406 18270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:26.170605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33429 (local address 127.0.97.1:46449)
0924 10:00:26.171460 (+ 855us) negotiation.cc:107] Waiting for socket to connect
0924 10:00:26.171499 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 10:00:26.173061 (+ 1562us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:00:26.173731 (+ 670us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:00:26.173743 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:00:26.173854 (+ 111us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:00:26.174419 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:00:26.174441 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:00:26.175595 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:00:26.175601 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:00:26.177500 (+ 1899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:00:26.177515 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:00:26.178237 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:00:26.178244 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:00:26.178513 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:00:26.179187 (+ 674us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 10:00:26.179190 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:00:26.179263 (+ 73us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 10:00:26.179360 (+ 97us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:00:26.179685 (+ 325us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:00:26.179694 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:00:26.179701 (+ 7us) client_negotiation.cc:769] Sending connection context
0924 10:00:26.179860 (+ 159us) client_negotiation.cc:240] Negotiation successful
0924 10:00:26.180232 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":245,"threads_started":1}
I20260924 10:00:26.182766 18268 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33429
I20260924 10:00:26.183044 18268 heartbeater.cc:464] Registering TS with master...
I20260924 10:00:26.183791 18268 heartbeater.cc:511] Master 127.0.97.62:33429 requested a full tablet report, sending...
I20260924 10:00:26.185868 18082 ts_manager.cc:194] Registered new tserver with Master: 23eab3890a774185a370ebe62fe5f762 (127.0.97.1:46719)
I20260924 10:00:26.187288 18082 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.1:46449
W20260924 10:00:26.487941 18271 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:00:26.488204 18271 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:00:26.488339 18271 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:00:26.495784 18271 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:00:26.495877 18271 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:00:26.496018 18271 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260924 10:00:26.505021 18271 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790243735634835-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790243735634835-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790243735634835-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33429
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790243735634835-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:00:26.506181 18271 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:00:26.507920 18271 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:00:26.520046 18279 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:00:26.520011 18276 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:00:26.520145 18277 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:00:26.521183 18271 server_base.cc:1034] running on GCE node
I20260924 10:00:26.521783 18271 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:00:26.522676 18271 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:00:26.523882 18271 hybrid_clock.cc:648] HybridClock initialized: now 1790244026523817 us; error 54 us; skew 500 ppm
I20260924 10:00:26.524188 18271 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 10:00:26.526363 18271 webserver.cc:460] Webserver started at http://127.0.97.2:34621/ using document root <none> and password file <none>
I20260924 10:00:26.527034 18271 fs_manager.cc:359] Metadata directory not provided
I20260924 10:00:26.527132 18271 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:00:26.527468 18271 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:00:26.530341 18271 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790243735634835-388-0/minicluster-data/ts-1/data/instance:
uuid: "f32684f4e48449c8994600445773f5bf"
format_stamp: "Formatted at 2026-09-24 10:00:26 on dist-test-slave-w8d2"
server_key: "55368babcc9e4b7ab902d99378590aab"
server_key_iv: "65dc2c35f16feddf9c3a0369045a9233"
server_key_version: "encryptionkey@0"
I20260924 10:00:26.531107 18271 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790243735634835-388-0/minicluster-data/ts-1/wal/instance:
uuid: "f32684f4e48449c8994600445773f5bf"
format_stamp: "Formatted at 2026-09-24 10:00:26 on dist-test-slave-w8d2"
server_key: "55368babcc9e4b7ab902d99378590aab"
server_key_iv: "65dc2c35f16feddf9c3a0369045a9233"
server_key_version: "encryptionkey@0"
I20260924 10:00:26.536067 18271 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260924 10:00:26.539265 18285 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:00:26.540479 18271 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260924 10:00:26.540630 18271 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790243735634835-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790243735634835-388-0/minicluster-data/ts-1/wal
uuid: "f32684f4e48449c8994600445773f5bf"
format_stamp: "Formatted at 2026-09-24 10:00:26 on dist-test-slave-w8d2"
server_key: "55368babcc9e4b7ab902d99378590aab"
server_key_iv: "65dc2c35f16feddf9c3a0369045a9233"
server_key_version: "encryptionkey@0"
I20260924 10:00:26.540812 18271 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790243735634835-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790243735634835-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790243735634835-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:00:26.565970 18271 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:00:26.566777 18271 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:00:26.567027 18271 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:00:26.567972 18271 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:00:26.569394 18271 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:00:26.569483 18271 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:00:26.569581 18271 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:00:26.569649 18271 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:00:26.597952 18271 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:43989
I20260924 10:00:26.598012 18397 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:43989 every 8 connection(s)
I20260924 10:00:26.599146 18271 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790243735634835-388-0/minicluster-data/ts-1/data/info.pb
I20260924 10:00:26.601660 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu as pid 18271
I20260924 10:00:26.601838 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790243735634835-388-0/minicluster-data/ts-1/wal/instance
I20260924 10:00:26.602365 388 external_mini_cluster.cc:1428] Setting key 7f1ca181e6b461509328f3b952732081
I20260924 10:00:26.605230 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
/tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790243735634835-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790243735634835-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790243735634835-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790243735634835-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33429
--builtin_ntp_servers=127.0.97.20:44485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 10:00:26.611537 18120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:26.601582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:50553 (local address 127.0.97.62:33429)
0924 10:00:26.601752 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0924 10:00:26.601758 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:00:26.603463 (+ 1705us) server_negotiation.cc:407] Connection header received
0924 10:00:26.605524 (+ 2061us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:00:26.605530 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:00:26.605610 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:00:26.605699 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:00:26.606713 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:00:26.607481 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:00:26.609005 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:00:26.609424 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:00:26.609529 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:00:26.610771 (+ 1242us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:00:26.610782 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:00:26.610785 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:00:26.610803 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:00:26.610835 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:00:26.610846 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:00:26.610916 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 10:00:26.611126 (+ 210us) server_negotiation.cc:299] Negotiation successful
0924 10:00:26.611223 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 10:00:26.612665 18400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:26.602257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33429 (local address 127.0.97.2:50553)
0924 10:00:26.603317 (+ 1060us) negotiation.cc:107] Waiting for socket to connect
0924 10:00:26.603365 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0924 10:00:26.605216 (+ 1851us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:00:26.605927 (+ 711us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:00:26.605935 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:00:26.606024 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:00:26.606558 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:00:26.606574 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:00:26.607613 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:00:26.607619 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:00:26.608869 (+ 1250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:00:26.608881 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:00:26.609623 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:00:26.609630 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:00:26.609855 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:00:26.610479 (+ 624us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 10:00:26.610482 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:00:26.610518 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 10:00:26.610592 (+ 74us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:00:26.610928 (+ 336us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:00:26.610939 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:00:26.610947 (+ 8us) client_negotiation.cc:769] Sending connection context
0924 10:00:26.611097 (+ 150us) client_negotiation.cc:240] Negotiation successful
0924 10:00:26.611503 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":274,"threads_started":1}
I20260924 10:00:26.613946 18398 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33429
I20260924 10:00:26.614228 18398 heartbeater.cc:464] Registering TS with master...
I20260924 10:00:26.615021 18398 heartbeater.cc:511] Master 127.0.97.62:33429 requested a full tablet report, sending...
I20260924 10:00:26.616453 18082 ts_manager.cc:194] Registered new tserver with Master: f32684f4e48449c8994600445773f5bf (127.0.97.2:43989)
I20260924 10:00:26.617348 18082 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.2:50553
W20260924 10:00:26.919746 18401 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:00:26.919996 18401 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:00:26.920112 18401 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:00:26.927538 18401 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:00:26.927634 18401 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:00:26.927780 18401 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260924 10:00:26.936542 18401 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790243735634835-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790243735634835-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790243735634835-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33429
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790243735634835-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:00:26.937774 18401 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:00:26.939481 18401 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:00:26.951722 18406 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:00:26.951756 18407 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:00:26.951812 18409 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:00:26.952800 18401 server_base.cc:1034] running on GCE node
I20260924 10:00:26.953383 18401 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:00:26.954298 18401 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:00:26.955493 18401 hybrid_clock.cc:648] HybridClock initialized: now 1790244026955441 us; error 39 us; skew 500 ppm
I20260924 10:00:26.955809 18401 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 10:00:26.957990 18401 webserver.cc:460] Webserver started at http://127.0.97.3:36219/ using document root <none> and password file <none>
I20260924 10:00:26.958678 18401 fs_manager.cc:359] Metadata directory not provided
I20260924 10:00:26.958776 18401 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:00:26.959079 18401 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:00:26.961903 18401 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790243735634835-388-0/minicluster-data/ts-2/data/instance:
uuid: "5add49f1e0bd483db30efcbde0618f43"
format_stamp: "Formatted at 2026-09-24 10:00:26 on dist-test-slave-w8d2"
server_key: "2392bc1c18a2ad4bffd83e9d8c4f14ed"
server_key_iv: "feded433462cd000de639a23bd21451d"
server_key_version: "encryptionkey@0"
I20260924 10:00:26.962646 18401 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790243735634835-388-0/minicluster-data/ts-2/wal/instance:
uuid: "5add49f1e0bd483db30efcbde0618f43"
format_stamp: "Formatted at 2026-09-24 10:00:26 on dist-test-slave-w8d2"
server_key: "2392bc1c18a2ad4bffd83e9d8c4f14ed"
server_key_iv: "feded433462cd000de639a23bd21451d"
server_key_version: "encryptionkey@0"
I20260924 10:00:26.967610 18401 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260924 10:00:26.970811 18415 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:00:26.972077 18401 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 10:00:26.972220 18401 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790243735634835-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790243735634835-388-0/minicluster-data/ts-2/wal
uuid: "5add49f1e0bd483db30efcbde0618f43"
format_stamp: "Formatted at 2026-09-24 10:00:26 on dist-test-slave-w8d2"
server_key: "2392bc1c18a2ad4bffd83e9d8c4f14ed"
server_key_iv: "feded433462cd000de639a23bd21451d"
server_key_version: "encryptionkey@0"
I20260924 10:00:26.972407 18401 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790243735634835-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790243735634835-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790243735634835-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:00:26.997064 18401 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:00:26.997900 18401 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:00:26.998143 18401 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:00:26.999045 18401 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:00:27.000530 18401 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:00:27.000626 18401 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:00:27.000727 18401 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:00:27.000798 18401 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:00:27.028913 18401 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:33749
I20260924 10:00:27.029001 18527 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:33749 every 8 connection(s)
I20260924 10:00:27.030124 18401 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790243735634835-388-0/minicluster-data/ts-2/data/info.pb
I20260924 10:00:27.037974 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu as pid 18401
I20260924 10:00:27.038146 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790243735634835-388-0/minicluster-data/ts-2/wal/instance
I20260924 10:00:27.038553 388 external_mini_cluster.cc:1428] Setting key 09b8963632888761d5f214b7a6653ec7
I20260924 10:00:27.042068 18120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:27.032526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:52225 (local address 127.0.97.62:33429)
0924 10:00:27.032715 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0924 10:00:27.032721 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:00:27.034135 (+ 1414us) server_negotiation.cc:407] Connection header received
0924 10:00:27.035903 (+ 1768us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:00:27.035908 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:00:27.035973 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:00:27.036054 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:00:27.037017 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:00:27.037724 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:00:27.039471 (+ 1747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:00:27.039932 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:00:27.040047 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:00:27.041324 (+ 1277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:00:27.041328 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:00:27.041331 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:00:27.041348 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:00:27.041380 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:00:27.041391 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:00:27.041461 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 10:00:27.041665 (+ 204us) server_negotiation.cc:299] Negotiation successful
0924 10:00:27.041760 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 10:00:27.042991 18530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:27.033120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33429 (local address 127.0.97.3:52225)
0924 10:00:27.033996 (+ 876us) negotiation.cc:107] Waiting for socket to connect
0924 10:00:27.034030 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0924 10:00:27.035657 (+ 1627us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:00:27.036230 (+ 573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:00:27.036238 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:00:27.036329 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:00:27.036874 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:00:27.036890 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:00:27.037870 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:00:27.037874 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:00:27.039330 (+ 1456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:00:27.039343 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:00:27.040115 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:00:27.040123 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:00:27.040399 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:00:27.041035 (+ 636us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 10:00:27.041038 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:00:27.041085 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 10:00:27.041162 (+ 77us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:00:27.041473 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:00:27.041483 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:00:27.041491 (+ 8us) client_negotiation.cc:769] Sending connection context
0924 10:00:27.041642 (+ 151us) client_negotiation.cc:240] Negotiation successful
0924 10:00:27.042017 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":258,"threads_started":1}
I20260924 10:00:27.044204 18528 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33429
I20260924 10:00:27.044483 18528 heartbeater.cc:464] Registering TS with master...
I20260924 10:00:27.045212 18528 heartbeater.cc:511] Master 127.0.97.62:33429 requested a full tablet report, sending...
I20260924 10:00:27.046413 18082 ts_manager.cc:194] Registered new tserver with Master: 5add49f1e0bd483db30efcbde0618f43 (127.0.97.3:33749)
I20260924 10:00:27.047151 18082 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.3:52225
I20260924 10:00:27.053022 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 10:00:27.062325 18120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:27.056719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43806 (local address 127.0.97.62:33429)
0924 10:00:27.056877 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0924 10:00:27.056882 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:00:27.057307 (+ 425us) server_negotiation.cc:407] Connection header received
0924 10:00:27.057453 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:00:27.057458 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:00:27.057559 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:00:27.057636 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:00:27.058103 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:00:27.058816 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:00:27.060232 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:00:27.060629 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:00:27.060732 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:00:27.061668 (+ 936us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:00:27.061674 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:00:27.061679 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:00:27.061697 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:00:27.061725 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:00:27.061736 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:00:27.061804 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 10:00:27.061928 (+ 124us) server_negotiation.cc:299] Negotiation successful
0924 10:00:27.062013 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 10:00:27.065579 18082 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43806:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 10:00:27.072314 18082 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 10:00:27.097821 18541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:27.083556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:43989 (local address 127.0.0.1:44888)
0924 10:00:27.084459 (+ 903us) negotiation.cc:107] Waiting for socket to connect
0924 10:00:27.084506 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0924 10:00:27.084700 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:00:27.088414 (+ 3714us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:00:27.088423 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:00:27.088470 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:00:27.088911 (+ 441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:00:27.088930 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:00:27.092563 (+ 3633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:00:27.092572 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:00:27.094892 (+ 2320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:00:27.094906 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:00:27.096839 (+ 1933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:00:27.096847 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:00:27.097059 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:00:27.097119 (+ 60us) client_negotiation.cc:769] Sending connection context
0924 10:00:27.097229 (+ 110us) client_negotiation.cc:240] Negotiation successful
0924 10:00:27.097412 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":13799,"threads_started":1}
I20260924 10:00:27.098737 18542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:27.083992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44888 (local address 127.0.97.2:43989)
0924 10:00:27.087732 (+ 3740us) server_negotiation.cc:206] Beginning negotiation
0924 10:00:27.087741 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 10:00:27.087775 (+ 34us) server_negotiation.cc:407] Connection header received
0924 10:00:27.087879 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:00:27.087888 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:00:27.088164 (+ 276us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:00:27.089131 (+ 967us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:00:27.089413 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:00:27.091086 (+ 1673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:00:27.095123 (+ 4037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:00:27.096632 (+ 1509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:00:27.097850 (+ 1218us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:00:27.098137 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0924 10:00:27.098208 (+ 71us) server_negotiation.cc:299] Negotiation successful
0924 10:00:27.098369 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3549,"thread_start_us":3409,"threads_started":1}
I20260924 10:00:27.099143 18544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:27.083633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:46719 (local address 127.0.0.1:56670)
0924 10:00:27.085344 (+ 1711us) negotiation.cc:107] Waiting for socket to connect
0924 10:00:27.085363 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 10:00:27.085494 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:00:27.092300 (+ 6806us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:00:27.092304 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:00:27.092327 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:00:27.092745 (+ 418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:00:27.092757 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:00:27.094651 (+ 1894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:00:27.094656 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:00:27.096749 (+ 2093us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:00:27.096763 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:00:27.098539 (+ 1776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:00:27.098546 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:00:27.098720 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:00:27.098752 (+ 32us) client_negotiation.cc:769] Sending connection context
0924 10:00:27.098815 (+ 63us) client_negotiation.cc:240] Negotiation successful
0924 10:00:27.098928 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1593,"thread_start_us":370,"threads_started":1}
I20260924 10:00:27.100119 18543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:27.084258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56670 (local address 127.0.97.1:46719)
0924 10:00:27.091844 (+ 7586us) server_negotiation.cc:206] Beginning negotiation
0924 10:00:27.091850 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:00:27.091873 (+ 23us) server_negotiation.cc:407] Connection header received
0924 10:00:27.091944 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:00:27.091950 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:00:27.092127 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:00:27.092285 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:00:27.092990 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:00:27.094510 (+ 1520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:00:27.097008 (+ 2498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:00:27.098325 (+ 1317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:00:27.099389 (+ 1064us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:00:27.099611 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0924 10:00:27.099660 (+ 49us) server_negotiation.cc:299] Negotiation successful
0924 10:00:27.099786 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7454,"thread_start_us":7052,"threads_started":1}
I20260924 10:00:27.102743 18333 tablet_service.cc:1467] Processing CreateTablet for tablet b971c2699e0d483384b1b83f1007111e (DEFAULT_TABLE table=test-table [id=0e32c69adf0544a3be2c61c32fe68e5d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:00:27.103487 18203 tablet_service.cc:1467] Processing CreateTablet for tablet b971c2699e0d483384b1b83f1007111e (DEFAULT_TABLE table=test-table [id=0e32c69adf0544a3be2c61c32fe68e5d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:00:27.104358 18333 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b971c2699e0d483384b1b83f1007111e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:00:27.104817 18203 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b971c2699e0d483384b1b83f1007111e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:00:27.106118 18540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:27.084809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:33749 (local address 127.0.0.1:45514)
0924 10:00:27.098664 (+ 13855us) negotiation.cc:107] Waiting for socket to connect
0924 10:00:27.098682 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 10:00:27.098793 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:00:27.099913 (+ 1120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:00:27.099917 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:00:27.099939 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:00:27.100349 (+ 410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:00:27.100362 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:00:27.102188 (+ 1826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:00:27.102193 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:00:27.103533 (+ 1340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:00:27.103544 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:00:27.105465 (+ 1921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:00:27.105471 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:00:27.105609 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:00:27.105631 (+ 22us) client_negotiation.cc:769] Sending connection context
0924 10:00:27.105706 (+ 75us) client_negotiation.cc:240] Negotiation successful
0924 10:00:27.105820 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12437,"thread_start_us":626,"threads_started":1}
I20260924 10:00:27.106437 18545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:27.085304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45514 (local address 127.0.97.3:33749)
0924 10:00:27.097680 (+ 12376us) server_negotiation.cc:206] Beginning negotiation
0924 10:00:27.097688 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 10:00:27.098779 (+ 1091us) server_negotiation.cc:407] Connection header received
0924 10:00:27.099492 (+ 713us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:00:27.099500 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:00:27.099739 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:00:27.099908 (+ 169us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:00:27.100504 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:00:27.102059 (+ 1555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:00:27.103705 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:00:27.105020 (+ 1315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:00:27.105157 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:00:27.105384 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0924 10:00:27.105759 (+ 375us) server_negotiation.cc:299] Negotiation successful
0924 10:00:27.105908 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12216,"thread_start_us":12391,"threads_started":1}
I20260924 10:00:27.109138 18463 tablet_service.cc:1467] Processing CreateTablet for tablet b971c2699e0d483384b1b83f1007111e (DEFAULT_TABLE table=test-table [id=0e32c69adf0544a3be2c61c32fe68e5d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:00:27.110105 18463 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b971c2699e0d483384b1b83f1007111e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:00:27.118505 18546 tablet_bootstrap.cc:492] T b971c2699e0d483384b1b83f1007111e P f32684f4e48449c8994600445773f5bf: Bootstrap starting.
I20260924 10:00:27.119284 18547 tablet_bootstrap.cc:492] T b971c2699e0d483384b1b83f1007111e P 23eab3890a774185a370ebe62fe5f762: Bootstrap starting.
I20260924 10:00:27.122161 18548 tablet_bootstrap.cc:492] T b971c2699e0d483384b1b83f1007111e P 5add49f1e0bd483db30efcbde0618f43: Bootstrap starting.
I20260924 10:00:27.122653 18546 tablet_bootstrap.cc:654] T b971c2699e0d483384b1b83f1007111e P f32684f4e48449c8994600445773f5bf: Neither blocks nor log segments found. Creating new log.
I20260924 10:00:27.123816 18546 log.cc:824] T b971c2699e0d483384b1b83f1007111e P f32684f4e48449c8994600445773f5bf: Log is configured to *not* fsync() on all Append() calls
I20260924 10:00:27.125746 18547 tablet_bootstrap.cc:654] T b971c2699e0d483384b1b83f1007111e P 23eab3890a774185a370ebe62fe5f762: Neither blocks nor log segments found. Creating new log.
I20260924 10:00:27.125746 18548 tablet_bootstrap.cc:654] T b971c2699e0d483384b1b83f1007111e P 5add49f1e0bd483db30efcbde0618f43: Neither blocks nor log segments found. Creating new log.
I20260924 10:00:27.126853 18547 log.cc:824] T b971c2699e0d483384b1b83f1007111e P 23eab3890a774185a370ebe62fe5f762: Log is configured to *not* fsync() on all Append() calls
I20260924 10:00:27.126858 18548 log.cc:824] T b971c2699e0d483384b1b83f1007111e P 5add49f1e0bd483db30efcbde0618f43: Log is configured to *not* fsync() on all Append() calls
I20260924 10:00:27.129979 18547 tablet_bootstrap.cc:492] T b971c2699e0d483384b1b83f1007111e P 23eab3890a774185a370ebe62fe5f762: No bootstrap required, opened a new log
I20260924 10:00:27.130013 18548 tablet_bootstrap.cc:492] T b971c2699e0d483384b1b83f1007111e P 5add49f1e0bd483db30efcbde0618f43: No bootstrap required, opened a new log
I20260924 10:00:27.130051 18546 tablet_bootstrap.cc:492] T b971c2699e0d483384b1b83f1007111e P f32684f4e48449c8994600445773f5bf: No bootstrap required, opened a new log
I20260924 10:00:27.130255 18547 ts_tablet_manager.cc:1397] T b971c2699e0d483384b1b83f1007111e P 23eab3890a774185a370ebe62fe5f762: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.004s
I20260924 10:00:27.130255 18548 ts_tablet_manager.cc:1397] T b971c2699e0d483384b1b83f1007111e P 5add49f1e0bd483db30efcbde0618f43: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.001s
I20260924 10:00:27.130268 18546 ts_tablet_manager.cc:1397] T b971c2699e0d483384b1b83f1007111e P f32684f4e48449c8994600445773f5bf: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.005s
I20260924 10:00:27.137069 18546 raft_consensus.cc:348] T b971c2699e0d483384b1b83f1007111e P f32684f4e48449c8994600445773f5bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f32684f4e48449c8994600445773f5bf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43989 } } peers { permanent_uuid: "23eab3890a774185a370ebe62fe5f762" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46719 } } peers { permanent_uuid: "5add49f1e0bd483db30efcbde0618f43" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33749 } }
I20260924 10:00:27.137394 18546 raft_consensus.cc:374] T b971c2699e0d483384b1b83f1007111e P f32684f4e48449c8994600445773f5bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:00:27.137480 18546 raft_consensus.cc:729] T b971c2699e0d483384b1b83f1007111e P f32684f4e48449c8994600445773f5bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f32684f4e48449c8994600445773f5bf, State: Initialized, Role: FOLLOWER
I20260924 10:00:27.138095 18546 consensus_queue.cc:260] T b971c2699e0d483384b1b83f1007111e P f32684f4e48449c8994600445773f5bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f32684f4e48449c8994600445773f5bf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43989 } } peers { permanent_uuid: "23eab3890a774185a370ebe62fe5f762" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46719 } } peers { permanent_uuid: "5add49f1e0bd483db30efcbde0618f43" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33749 } }
I20260924 10:00:27.139436 18398 heartbeater.cc:503] Master 127.0.97.62:33429 was elected leader, sending a full tablet report...
I20260924 10:00:27.140103 18546 ts_tablet_manager.cc:1428] T b971c2699e0d483384b1b83f1007111e P f32684f4e48449c8994600445773f5bf: Time spent starting tablet: real 0.010s user 0.011s sys 0.000s
I20260924 10:00:27.140228 18547 raft_consensus.cc:348] T b971c2699e0d483384b1b83f1007111e P 23eab3890a774185a370ebe62fe5f762 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f32684f4e48449c8994600445773f5bf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43989 } } peers { permanent_uuid: "23eab3890a774185a370ebe62fe5f762" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46719 } } peers { permanent_uuid: "5add49f1e0bd483db30efcbde0618f43" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33749 } }
I20260924 10:00:27.140228 18548 raft_consensus.cc:348] T b971c2699e0d483384b1b83f1007111e P 5add49f1e0bd483db30efcbde0618f43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f32684f4e48449c8994600445773f5bf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43989 } } peers { permanent_uuid: "23eab3890a774185a370ebe62fe5f762" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46719 } } peers { permanent_uuid: "5add49f1e0bd483db30efcbde0618f43" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33749 } }
I20260924 10:00:27.140612 18548 raft_consensus.cc:374] T b971c2699e0d483384b1b83f1007111e P 5add49f1e0bd483db30efcbde0618f43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:00:27.140615 18547 raft_consensus.cc:374] T b971c2699e0d483384b1b83f1007111e P 23eab3890a774185a370ebe62fe5f762 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:00:27.140712 18547 raft_consensus.cc:729] T b971c2699e0d483384b1b83f1007111e P 23eab3890a774185a370ebe62fe5f762 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eab3890a774185a370ebe62fe5f762, State: Initialized, Role: FOLLOWER
I20260924 10:00:27.140717 18548 raft_consensus.cc:729] T b971c2699e0d483384b1b83f1007111e P 5add49f1e0bd483db30efcbde0618f43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5add49f1e0bd483db30efcbde0618f43, State: Initialized, Role: FOLLOWER
I20260924 10:00:27.141270 18548 consensus_queue.cc:260] T b971c2699e0d483384b1b83f1007111e P 5add49f1e0bd483db30efcbde0618f43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f32684f4e48449c8994600445773f5bf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43989 } } peers { permanent_uuid: "23eab3890a774185a370ebe62fe5f762" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46719 } } peers { permanent_uuid: "5add49f1e0bd483db30efcbde0618f43" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33749 } }
I20260924 10:00:27.141270 18547 consensus_queue.cc:260] T b971c2699e0d483384b1b83f1007111e P 23eab3890a774185a370ebe62fe5f762 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f32684f4e48449c8994600445773f5bf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43989 } } peers { permanent_uuid: "23eab3890a774185a370ebe62fe5f762" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46719 } } peers { permanent_uuid: "5add49f1e0bd483db30efcbde0618f43" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33749 } }
I20260924 10:00:27.142656 18528 heartbeater.cc:503] Master 127.0.97.62:33429 was elected leader, sending a full tablet report...
I20260924 10:00:27.142720 18268 heartbeater.cc:503] Master 127.0.97.62:33429 was elected leader, sending a full tablet report...
I20260924 10:00:27.143193 18548 ts_tablet_manager.cc:1428] T b971c2699e0d483384b1b83f1007111e P 5add49f1e0bd483db30efcbde0618f43: Time spent starting tablet: real 0.013s user 0.011s sys 0.003s
I20260924 10:00:27.143896 18547 ts_tablet_manager.cc:1428] T b971c2699e0d483384b1b83f1007111e P 23eab3890a774185a370ebe62fe5f762: Time spent starting tablet: real 0.014s user 0.008s sys 0.006s
W20260924 10:00:27.169665 18269 tablet.cc:2369] T b971c2699e0d483384b1b83f1007111e P 23eab3890a774185a370ebe62fe5f762: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 10:00:27.281684 18529 tablet.cc:2369] T b971c2699e0d483384b1b83f1007111e P 5add49f1e0bd483db30efcbde0618f43: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 10:00:27.351084 18399 tablet.cc:2369] T b971c2699e0d483384b1b83f1007111e P f32684f4e48449c8994600445773f5bf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 10:00:27.395877 18554 raft_consensus.cc:482] T b971c2699e0d483384b1b83f1007111e P 5add49f1e0bd483db30efcbde0618f43 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 10:00:27.396225 18554 raft_consensus.cc:504] T b971c2699e0d483384b1b83f1007111e P 5add49f1e0bd483db30efcbde0618f43 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f32684f4e48449c8994600445773f5bf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43989 } } peers { permanent_uuid: "23eab3890a774185a370ebe62fe5f762" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46719 } } peers { permanent_uuid: "5add49f1e0bd483db30efcbde0618f43" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33749 } }
I20260924 10:00:27.397926 18554 leader_election.cc:288] T b971c2699e0d483384b1b83f1007111e P 5add49f1e0bd483db30efcbde0618f43 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f32684f4e48449c8994600445773f5bf (127.0.97.2:43989), 23eab3890a774185a370ebe62fe5f762 (127.0.97.1:46719)
I20260924 10:00:27.403442 18543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:27.398495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:51861 (local address 127.0.97.1:46719)
0924 10:00:27.398661 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0924 10:00:27.398668 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:00:27.398688 (+ 20us) server_negotiation.cc:407] Connection header received
0924 10:00:27.398763 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:00:27.398767 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:00:27.398865 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:00:27.398964 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:00:27.399479 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:00:27.400167 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:00:27.401880 (+ 1713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:00:27.402620 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:00:27.402728 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:00:27.402814 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0924 10:00:27.403056 (+ 242us) server_negotiation.cc:299] Negotiation successful
0924 10:00:27.403177 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260924 10:00:27.403481 18530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:27.398223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:46719 (local address 127.0.97.3:51861)
0924 10:00:27.398479 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0924 10:00:27.398498 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 10:00:27.398616 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:00:27.399035 (+ 419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:00:27.399039 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:00:27.399124 (+ 85us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:00:27.399330 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:00:27.399339 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:00:27.400292 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:00:27.400297 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:00:27.401717 (+ 1420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:00:27.401727 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:00:27.402761 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:00:27.402767 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:00:27.402919 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:00:27.402941 (+ 22us) client_negotiation.cc:769] Sending connection context
0924 10:00:27.403027 (+ 86us) client_negotiation.cc:240] Negotiation successful
0924 10:00:27.403160 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"mutex_wait_us":19,"spinlock_wait_cycles":24960}
I20260924 10:00:27.403750 18542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:27.398469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:33011 (local address 127.0.97.2:43989)
0924 10:00:27.398665 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0924 10:00:27.398670 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:00:27.399033 (+ 363us) server_negotiation.cc:407] Connection header received
0924 10:00:27.399213 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:00:27.399217 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:00:27.399306 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:00:27.399379 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:00:27.399858 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:00:27.400506 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:00:27.402180 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:00:27.402923 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:00:27.403026 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:00:27.403114 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0924 10:00:27.403398 (+ 284us) server_negotiation.cc:299] Negotiation successful
0924 10:00:27.403520 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260924 10:00:27.404031 18555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:27.398260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:43989 (local address 127.0.97.3:33011)
0924 10:00:27.398938 (+ 678us) negotiation.cc:107] Waiting for socket to connect
0924 10:00:27.398955 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 10:00:27.399070 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:00:27.399405 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:00:27.399409 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:00:27.399430 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:00:27.399714 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:00:27.399722 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:00:27.400643 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:00:27.400649 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:00:27.402015 (+ 1366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:00:27.402026 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:00:27.403068 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:00:27.403074 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:00:27.403220 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:00:27.403278 (+ 58us) client_negotiation.cc:769] Sending connection context
0924 10:00:27.403398 (+ 120us) client_negotiation.cc:240] Negotiation successful
0924 10:00:27.403673 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":570,"mutex_wait_us":1,"thread_start_us":785,"threads_started":1}
I20260924 10:00:27.404448 18223 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b971c2699e0d483384b1b83f1007111e" candidate_uuid: "5add49f1e0bd483db30efcbde0618f43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eab3890a774185a370ebe62fe5f762" is_pre_election: true
I20260924 10:00:27.405011 18223 raft_consensus.cc:2454] T b971c2699e0d483384b1b83f1007111e P 23eab3890a774185a370ebe62fe5f762 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5add49f1e0bd483db30efcbde0618f43 in term 0.
I20260924 10:00:27.404969 18353 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b971c2699e0d483384b1b83f1007111e" candidate_uuid: "5add49f1e0bd483db30efcbde0618f43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f32684f4e48449c8994600445773f5bf" is_pre_election: true
I20260924 10:00:27.405517 18353 raft_consensus.cc:2454] T b971c2699e0d483384b1b83f1007111e P f32684f4e48449c8994600445773f5bf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5add49f1e0bd483db30efcbde0618f43 in term 0.
I20260924 10:00:27.405622 18419 leader_election.cc:302] T b971c2699e0d483384b1b83f1007111e P 5add49f1e0bd483db30efcbde0618f43 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23eab3890a774185a370ebe62fe5f762, 5add49f1e0bd483db30efcbde0618f43; no voters:
I20260924 10:00:27.405969 18554 raft_consensus.cc:2781] T b971c2699e0d483384b1b83f1007111e P 5add49f1e0bd483db30efcbde0618f43 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 10:00:27.406083 18554 raft_consensus.cc:482] T b971c2699e0d483384b1b83f1007111e P 5add49f1e0bd483db30efcbde0618f43 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 10:00:27.406180 18554 raft_consensus.cc:3037] T b971c2699e0d483384b1b83f1007111e P 5add49f1e0bd483db30efcbde0618f43 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:00:27.408283 18554 raft_consensus.cc:504] T b971c2699e0d483384b1b83f1007111e P 5add49f1e0bd483db30efcbde0618f43 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f32684f4e48449c8994600445773f5bf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43989 } } peers { permanent_uuid: "23eab3890a774185a370ebe62fe5f762" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46719 } } peers { permanent_uuid: "5add49f1e0bd483db30efcbde0618f43" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33749 } }
I20260924 10:00:27.408842 18554 leader_election.cc:288] T b971c2699e0d483384b1b83f1007111e P 5add49f1e0bd483db30efcbde0618f43 [CANDIDATE]: Term 1 election: Requested vote from peers f32684f4e48449c8994600445773f5bf (127.0.97.2:43989), 23eab3890a774185a370ebe62fe5f762 (127.0.97.1:46719)
I20260924 10:00:27.409161 18353 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b971c2699e0d483384b1b83f1007111e" candidate_uuid: "5add49f1e0bd483db30efcbde0618f43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f32684f4e48449c8994600445773f5bf"
I20260924 10:00:27.409281 18223 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b971c2699e0d483384b1b83f1007111e" candidate_uuid: "5add49f1e0bd483db30efcbde0618f43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eab3890a774185a370ebe62fe5f762"
I20260924 10:00:27.409353 18353 raft_consensus.cc:3037] T b971c2699e0d483384b1b83f1007111e P f32684f4e48449c8994600445773f5bf [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:00:27.409473 18223 raft_consensus.cc:3037] T b971c2699e0d483384b1b83f1007111e P 23eab3890a774185a370ebe62fe5f762 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:00:27.412406 18223 raft_consensus.cc:2454] T b971c2699e0d483384b1b83f1007111e P 23eab3890a774185a370ebe62fe5f762 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5add49f1e0bd483db30efcbde0618f43 in term 1.
I20260924 10:00:27.412406 18353 raft_consensus.cc:2454] T b971c2699e0d483384b1b83f1007111e P f32684f4e48449c8994600445773f5bf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5add49f1e0bd483db30efcbde0618f43 in term 1.
I20260924 10:00:27.412813 18417 leader_election.cc:302] T b971c2699e0d483384b1b83f1007111e P 5add49f1e0bd483db30efcbde0618f43 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 23eab3890a774185a370ebe62fe5f762, 5add49f1e0bd483db30efcbde0618f43, f32684f4e48449c8994600445773f5bf; no voters:
I20260924 10:00:27.413175 18554 raft_consensus.cc:2781] T b971c2699e0d483384b1b83f1007111e P 5add49f1e0bd483db30efcbde0618f43 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:00:27.413579 18554 raft_consensus.cc:686] T b971c2699e0d483384b1b83f1007111e P 5add49f1e0bd483db30efcbde0618f43 [term 1 LEADER]: Becoming Leader. State: Replica: 5add49f1e0bd483db30efcbde0618f43, State: Running, Role: LEADER
I20260924 10:00:27.414024 18554 consensus_queue.cc:237] T b971c2699e0d483384b1b83f1007111e P 5add49f1e0bd483db30efcbde0618f43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f32684f4e48449c8994600445773f5bf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43989 } } peers { permanent_uuid: "23eab3890a774185a370ebe62fe5f762" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46719 } } peers { permanent_uuid: "5add49f1e0bd483db30efcbde0618f43" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33749 } }
I20260924 10:00:27.418562 18081 catalog_manager.cc:5495] T b971c2699e0d483384b1b83f1007111e P 5add49f1e0bd483db30efcbde0618f43 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5add49f1e0bd483db30efcbde0618f43 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "5add49f1e0bd483db30efcbde0618f43" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f32684f4e48449c8994600445773f5bf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43989 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "23eab3890a774185a370ebe62fe5f762" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46719 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5add49f1e0bd483db30efcbde0618f43" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33749 } health_report { overall_health: HEALTHY } } }
I20260924 10:00:27.472373 18545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:27.467452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45518 (local address 127.0.97.3:33749)
0924 10:00:27.467645 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0924 10:00:27.467649 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:00:27.467664 (+ 15us) server_negotiation.cc:407] Connection header received
0924 10:00:27.467746 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:00:27.467750 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:00:27.467821 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:00:27.467949 (+ 128us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:00:27.468282 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:00:27.468997 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:00:27.470319 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:00:27.470716 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:00:27.470819 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:00:27.471305 (+ 486us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:00:27.471836 (+ 531us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:00:27.471966 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0924 10:00:27.472057 (+ 91us) server_negotiation.cc:299] Negotiation successful
0924 10:00:27.472136 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 10:00:27.479739 18223 raft_consensus.cc:1261] T b971c2699e0d483384b1b83f1007111e P 23eab3890a774185a370ebe62fe5f762 [term 1 FOLLOWER]: Refusing update from remote peer 5add49f1e0bd483db30efcbde0618f43: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:00:27.479741 18353 raft_consensus.cc:1261] T b971c2699e0d483384b1b83f1007111e P f32684f4e48449c8994600445773f5bf [term 1 FOLLOWER]: Refusing update from remote peer 5add49f1e0bd483db30efcbde0618f43: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:00:27.480638 18556 consensus_queue.cc:1035] T b971c2699e0d483384b1b83f1007111e P 5add49f1e0bd483db30efcbde0618f43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23eab3890a774185a370ebe62fe5f762" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:00:27.480928 18554 consensus_queue.cc:1035] T b971c2699e0d483384b1b83f1007111e P 5add49f1e0bd483db30efcbde0618f43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f32684f4e48449c8994600445773f5bf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:00:27.503077 18080 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:43806:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:00:27.503360 18080 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:43806:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:00:27.506690 18080 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 929d96eb44f74ae2b1ffca93d2f5eb4f: Sending DeleteTablet for 3 replicas of tablet b971c2699e0d483384b1b83f1007111e
I20260924 10:00:27.507850 18203 tablet_service.cc:1514] Processing DeleteTablet for tablet b971c2699e0d483384b1b83f1007111e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:00:27 UTC) from {username='slave'} at 127.0.0.1:56670
I20260924 10:00:27.508087 18333 tablet_service.cc:1514] Processing DeleteTablet for tablet b971c2699e0d483384b1b83f1007111e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:00:27 UTC) from {username='slave'} at 127.0.0.1:44888
I20260924 10:00:27.508108 18463 tablet_service.cc:1514] Processing DeleteTablet for tablet b971c2699e0d483384b1b83f1007111e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:00:27 UTC) from {username='slave'} at 127.0.0.1:45514
I20260924 10:00:27.509819 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu pid 18141
I20260924 10:00:27.539983 18571 tablet_replica.cc:330] T b971c2699e0d483384b1b83f1007111e P f32684f4e48449c8994600445773f5bf: stopping tablet replica
I20260924 10:00:27.540546 18571 raft_consensus.cc:2229] T b971c2699e0d483384b1b83f1007111e P f32684f4e48449c8994600445773f5bf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:00:27.541178 18571 raft_consensus.cc:2258] T b971c2699e0d483384b1b83f1007111e P f32684f4e48449c8994600445773f5bf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:00:27.545753 18571 ts_tablet_manager.cc:1905] T b971c2699e0d483384b1b83f1007111e P f32684f4e48449c8994600445773f5bf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:00:27.548544 18570 tablet_replica.cc:330] T b971c2699e0d483384b1b83f1007111e P 5add49f1e0bd483db30efcbde0618f43: stopping tablet replica
I20260924 10:00:27.549077 18570 raft_consensus.cc:2229] T b971c2699e0d483384b1b83f1007111e P 5add49f1e0bd483db30efcbde0618f43 [term 1 LEADER]: Raft consensus shutting down.
I20260924 10:00:27.549875 18570 raft_consensus.cc:2258] T b971c2699e0d483384b1b83f1007111e P 5add49f1e0bd483db30efcbde0618f43 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:00:27.550783 18543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:27.511226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56684 (local address 127.0.97.1:46719)
0924 10:00:27.511419 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0924 10:00:27.511425 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:00:27.534949 (+ 23524us) server_negotiation.cc:407] Connection header received
0924 10:00:27.535517 (+ 568us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:00:27.535523 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:00:27.535639 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:00:27.535718 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:00:27.536437 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:00:27.537643 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:00:27.541406 (+ 3763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:00:27.542062 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:00:27.542155 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:00:27.546622 (+ 4467us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:00:27.546633 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:00:27.546650 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:00:27.547424 (+ 774us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:00:27.547486 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:00:27.547509 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:00:27.547588 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0924 10:00:27.550307 (+ 2719us) server_negotiation.cc:299] Negotiation successful
0924 10:00:27.550438 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 10:00:27.552162 18263 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:00:27.552143 18570 ts_tablet_manager.cc:1905] T b971c2699e0d483384b1b83f1007111e P 5add49f1e0bd483db30efcbde0618f43: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:00:27.554234 18571 ts_tablet_manager.cc:1918] T b971c2699e0d483384b1b83f1007111e P f32684f4e48449c8994600445773f5bf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:00:27.554399 18571 log.cc:1196] T b971c2699e0d483384b1b83f1007111e P f32684f4e48449c8994600445773f5bf: Deleting WAL directory at /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790243735634835-388-0/minicluster-data/ts-1/wal/wals/b971c2699e0d483384b1b83f1007111e
I20260924 10:00:27.554944 18571 ts_tablet_manager.cc:1939] T b971c2699e0d483384b1b83f1007111e P f32684f4e48449c8994600445773f5bf: Deleting consensus metadata
I20260924 10:00:27.557402 18067 catalog_manager.cc:4841] TS f32684f4e48449c8994600445773f5bf (127.0.97.2:43989): tablet b971c2699e0d483384b1b83f1007111e (table test-table [id=0e32c69adf0544a3be2c61c32fe68e5d]) successfully deleted
I20260924 10:00:27.559391 18570 ts_tablet_manager.cc:1918] T b971c2699e0d483384b1b83f1007111e P 5add49f1e0bd483db30efcbde0618f43: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:00:27.559571 18570 log.cc:1196] T b971c2699e0d483384b1b83f1007111e P 5add49f1e0bd483db30efcbde0618f43: Deleting WAL directory at /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790243735634835-388-0/minicluster-data/ts-2/wal/wals/b971c2699e0d483384b1b83f1007111e
I20260924 10:00:27.559993 18570 ts_tablet_manager.cc:1939] T b971c2699e0d483384b1b83f1007111e P 5add49f1e0bd483db30efcbde0618f43: Deleting consensus metadata
I20260924 10:00:27.561296 18069 catalog_manager.cc:4841] TS 5add49f1e0bd483db30efcbde0618f43 (127.0.97.3:33749): tablet b971c2699e0d483384b1b83f1007111e (table test-table [id=0e32c69adf0544a3be2c61c32fe68e5d]) successfully deleted
I20260924 10:00:27.687374 18569 tablet_replica.cc:330] T b971c2699e0d483384b1b83f1007111e P 23eab3890a774185a370ebe62fe5f762: stopping tablet replica
I20260924 10:00:27.687635 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu with pid 18141
W20260924 10:00:27.701869 18069 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:46719 (error 108)
I20260924 10:00:27.702117 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu pid 18271
I20260924 10:00:27.703981 18575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:27.703164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:46719 (local address 127.0.0.1:56686)
0924 10:00:27.703644 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0924 10:00:27.703785 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:46719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":234,"threads_started":1}
W20260924 10:00:27.704253 18069 catalog_manager.cc:4584] TS 23eab3890a774185a370ebe62fe5f762 (127.0.97.1:46719): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b971c2699e0d483384b1b83f1007111e: Network error: Client connection negotiation failed: client connection to 127.0.97.1:46719: connect: Connection refused (error 111)
I20260924 10:00:27.709851 18542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:27.702897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44900 (local address 127.0.97.2:43989)
0924 10:00:27.703083 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0924 10:00:27.703089 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:00:27.703110 (+ 21us) server_negotiation.cc:407] Connection header received
0924 10:00:27.703314 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:00:27.703320 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:00:27.703432 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:00:27.703534 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:00:27.704050 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:00:27.705171 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:00:27.706497 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:00:27.707156 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:00:27.707331 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:00:27.708357 (+ 1026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:00:27.708369 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:00:27.708386 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:00:27.709144 (+ 758us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:00:27.709201 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:00:27.709238 (+ 37us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:00:27.709350 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0924 10:00:27.709400 (+ 50us) server_negotiation.cc:299] Negotiation successful
0924 10:00:27.709506 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 10:00:27.710569 18393 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:00:27.757994 18575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:27.757480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:46719 (local address 127.0.0.1:56696)
0924 10:00:27.757716 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0924 10:00:27.757827 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:46719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260924 10:00:27.759209 18575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:27.758768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:46719 (local address 127.0.0.1:56700)
0924 10:00:27.758963 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0924 10:00:27.759045 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:46719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260924 10:00:27.797317 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu with pid 18271
I20260924 10:00:27.811496 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu pid 18401
I20260924 10:00:27.818171 18545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:27.812321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45520 (local address 127.0.97.3:33749)
0924 10:00:27.812524 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0924 10:00:27.812529 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:00:27.812546 (+ 17us) server_negotiation.cc:407] Connection header received
0924 10:00:27.812601 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:00:27.812605 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:00:27.812700 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:00:27.812795 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:00:27.813192 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:00:27.813982 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:00:27.815264 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:00:27.815703 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:00:27.815809 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:00:27.816845 (+ 1036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:00:27.816854 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:00:27.816871 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:00:27.817457 (+ 586us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:00:27.817512 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:00:27.817531 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:00:27.817620 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0924 10:00:27.817764 (+ 144us) server_negotiation.cc:299] Negotiation successful
0924 10:00:27.817859 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260924 10:00:27.818812 18523 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:00:27.819527 18575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:27.819094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:46719 (local address 127.0.0.1:56712)
0924 10:00:27.819317 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0924 10:00:27.819391 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:46719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260924 10:00:27.820641 18575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:27.820255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:46719 (local address 127.0.0.1:56720)
0924 10:00:27.820401 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0924 10:00:27.820479 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:46719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260924 10:00:27.901146 18575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:27.900620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:46719 (local address 127.0.0.1:56730)
0924 10:00:27.900858 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0924 10:00:27.900971 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:46719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260924 10:00:27.902338 18575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:27.901888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:46719 (local address 127.0.0.1:56734)
0924 10:00:27.902098 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0924 10:00:27.902185 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:46719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106}
I20260924 10:00:27.935300 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu with pid 18401
I20260924 10:00:27.950356 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu pid 18051
I20260924 10:00:27.951493 18112 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:00:28.058642 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu with pid 18051
2026-09-24T10:00:28Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (2865 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790243735634835-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:00:28 dist-test-slave-w8d2 krb5kdc[18581](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:00:28 dist-test-slave-w8d2 krb5kdc[18581](info): set up 2 sockets
Sep 24 10:00:28 dist-test-slave-w8d2 krb5kdc[18581](info): commencing operation
krb5kdc: starting...
W20260924 10:00:29.063012 388 mini_kdc.cc:121] Time spent starting KDC: real 0.965s user 0.000s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790243735634835-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790243735634835-388-0/krb5kdc/test-user.keytab.
Sep 24 10:00:29 dist-test-slave-w8d2 krb5kdc[18581](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244029, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:00:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:00:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790243735634835-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790243735634835-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260924 10:00:29.285333 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
/tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790243735634835-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790243735634835-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790243735634835-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790243735634835-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45923
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790243735634835-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45923
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790243735634835-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790243735634835-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790243735634835-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:00:29.592494 18594 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:00:29.592767 18594 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:00:29.592870 18594 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:00:29.599923 18594 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:00:29.600018 18594 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:00:29.600069 18594 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:00:29.600123 18594 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:00:29.600164 18594 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:00:29.609709 18594 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790243735634835-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790243735634835-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45923
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790243735634835-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45923
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790243735634835-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790243735634835-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:00:29.610886 18594 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:00:29.612125 18594 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:00:29.623588 18599 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:00:29.623498 18602 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:00:29.623535 18600 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:00:29.624336 18594 server_base.cc:1034] running on GCE node
I20260924 10:00:29.625090 18594 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:00:29.626494 18594 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:00:29.627692 18594 hybrid_clock.cc:648] HybridClock initialized: now 1790244029627638 us; error 43 us; skew 500 ppm
I20260924 10:00:29.628022 18594 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:00:29 dist-test-slave-w8d2 krb5kdc[18581](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244029, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:00:29.630879 18594 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260924 10:00:29.632256 18594 webserver.cc:460] Webserver started at http://127.0.97.62:35879/ using document root <none> and password file <none>
I20260924 10:00:29.632920 18594 fs_manager.cc:359] Metadata directory not provided
I20260924 10:00:29.633026 18594 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:00:29.633320 18594 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:00:29.636191 18594 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790243735634835-388-0/minicluster-data/master-0/data/instance:
uuid: "1eefa496903e4b40a841f91d78b12674"
format_stamp: "Formatted at 2026-09-24 10:00:29 on dist-test-slave-w8d2"
server_key: "1efed67f653aed7fc5109f96a2db6601"
server_key_iv: "a0a54501bb2bb998db0a8c7cf8b62575"
server_key_version: "encryptionkey@0"
I20260924 10:00:29.636893 18594 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790243735634835-388-0/minicluster-data/master-0/wal/instance:
uuid: "1eefa496903e4b40a841f91d78b12674"
format_stamp: "Formatted at 2026-09-24 10:00:29 on dist-test-slave-w8d2"
server_key: "1efed67f653aed7fc5109f96a2db6601"
server_key_iv: "a0a54501bb2bb998db0a8c7cf8b62575"
server_key_version: "encryptionkey@0"
I20260924 10:00:29.641672 18594 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 10:00:29.644860 18609 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:00:29.646080 18594 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 10:00:29.646225 18594 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790243735634835-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790243735634835-388-0/minicluster-data/master-0/wal
uuid: "1eefa496903e4b40a841f91d78b12674"
format_stamp: "Formatted at 2026-09-24 10:00:29 on dist-test-slave-w8d2"
server_key: "1efed67f653aed7fc5109f96a2db6601"
server_key_iv: "a0a54501bb2bb998db0a8c7cf8b62575"
server_key_version: "encryptionkey@0"
I20260924 10:00:29.646400 18594 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790243735634835-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790243735634835-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790243735634835-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:00:29.665920 18594 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:00:29.669683 18594 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:00:29.669937 18594 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:00:29.686532 18594 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:45923
I20260924 10:00:29.686561 18660 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:45923 every 8 connection(s)
I20260924 10:00:29.687918 18594 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790243735634835-388-0/minicluster-data/master-0/data/info.pb
I20260924 10:00:29.691452 18661 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:00:29.693840 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu as pid 18594
I20260924 10:00:29.694008 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790243735634835-388-0/minicluster-data/master-0/wal/instance
I20260924 10:00:29.694420 388 external_mini_cluster.cc:1428] Setting key 34d4fc554f10c755ef3ab5bc88f14c2b
I20260924 10:00:29.703688 18661 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1eefa496903e4b40a841f91d78b12674: Bootstrap starting.
I20260924 10:00:29.706954 18661 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1eefa496903e4b40a841f91d78b12674: Neither blocks nor log segments found. Creating new log.
I20260924 10:00:29.708191 18661 log.cc:824] T 00000000000000000000000000000000 P 1eefa496903e4b40a841f91d78b12674: Log is configured to *not* fsync() on all Append() calls
Sep 24 10:00:29 dist-test-slave-w8d2 krb5kdc[18581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244029, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 10:00:29.711163 18661 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1eefa496903e4b40a841f91d78b12674: No bootstrap required, opened a new log
I20260924 10:00:29.717289 18664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:29.696229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54292 (local address 127.0.97.62:45923)
0924 10:00:29.697011 (+ 782us) server_negotiation.cc:206] Beginning negotiation
0924 10:00:29.697026 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 10:00:29.697075 (+ 49us) server_negotiation.cc:407] Connection header received
0924 10:00:29.699308 (+ 2233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:00:29.699336 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:00:29.699810 (+ 474us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:00:29.700260 (+ 450us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:00:29.701446 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:00:29.703326 (+ 1880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:00:29.704671 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:00:29.705356 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:00:29.705497 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:00:29.710439 (+ 4942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:00:29.710447 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:00:29.710461 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:00:29.710494 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:00:29.713410 (+ 2916us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:00:29.714128 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:00:29.714134 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:00:29.714139 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:00:29.714299 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:00:29.714736 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:00:29.714739 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:00:29.714741 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:00:29.715285 (+ 544us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:00:29.715507 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0924 10:00:29.715936 (+ 429us) server_negotiation.cc:299] Negotiation successful
0924 10:00:29.716294 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":512,"thread_start_us":303,"threads_started":1}
I20260924 10:00:29.719602 18661 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1eefa496903e4b40a841f91d78b12674 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1eefa496903e4b40a841f91d78b12674" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45923 } }
I20260924 10:00:29.719992 18661 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1eefa496903e4b40a841f91d78b12674 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:00:29.720074 18661 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1eefa496903e4b40a841f91d78b12674 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1eefa496903e4b40a841f91d78b12674, State: Initialized, Role: FOLLOWER
I20260924 10:00:29.720554 18661 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1eefa496903e4b40a841f91d78b12674 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1eefa496903e4b40a841f91d78b12674" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45923 } }
I20260924 10:00:29.720753 18661 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1eefa496903e4b40a841f91d78b12674 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:00:29.720851 18661 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1eefa496903e4b40a841f91d78b12674 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:00:29.720964 18661 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1eefa496903e4b40a841f91d78b12674 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:00:29.723167 18661 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1eefa496903e4b40a841f91d78b12674 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1eefa496903e4b40a841f91d78b12674" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45923 } }
I20260924 10:00:29.723621 18661 leader_election.cc:302] T 00000000000000000000000000000000 P 1eefa496903e4b40a841f91d78b12674 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1eefa496903e4b40a841f91d78b12674; no voters:
I20260924 10:00:29.724121 18661 leader_election.cc:288] T 00000000000000000000000000000000 P 1eefa496903e4b40a841f91d78b12674 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:00:29.724218 18666 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1eefa496903e4b40a841f91d78b12674 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:00:29.724807 18666 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1eefa496903e4b40a841f91d78b12674 [term 1 LEADER]: Becoming Leader. State: Replica: 1eefa496903e4b40a841f91d78b12674, State: Running, Role: LEADER
I20260924 10:00:29.725282 18666 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1eefa496903e4b40a841f91d78b12674 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1eefa496903e4b40a841f91d78b12674" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45923 } }
I20260924 10:00:29.726122 18661 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1eefa496903e4b40a841f91d78b12674 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:00:29.727835 18667 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1eefa496903e4b40a841f91d78b12674 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1eefa496903e4b40a841f91d78b12674" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1eefa496903e4b40a841f91d78b12674" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45923 } } }
I20260924 10:00:29.728152 18667 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1eefa496903e4b40a841f91d78b12674 [sys.catalog]: This master's current role is: LEADER
I20260924 10:00:29.729624 18673 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:00:29.729584 18668 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1eefa496903e4b40a841f91d78b12674 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1eefa496903e4b40a841f91d78b12674. Latest consensus state: current_term: 1 leader_uuid: "1eefa496903e4b40a841f91d78b12674" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1eefa496903e4b40a841f91d78b12674" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45923 } } }
I20260924 10:00:29.729938 18668 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1eefa496903e4b40a841f91d78b12674 [sys.catalog]: This master's current role is: LEADER
I20260924 10:00:29.734616 18673 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:00:29.743609 18673 catalog_manager.cc:1347] Generated new cluster ID: 96b0de90af414588b9af54b9de299e61
I20260924 10:00:29.743736 18673 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:00:29.760149 18673 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:00:29.761386 18673 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:00:29.774072 18673 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1eefa496903e4b40a841f91d78b12674: Generated new TSK 0
I20260924 10:00:29.775197 18673 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790243735634835-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790243735634835-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260924 10:00:29.836573 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
/tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790243735634835-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790243735634835-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790243735634835-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790243735634835-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45923
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790243735634835-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790243735634835-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790243735634835-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790243735634835-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:00:30.153002 18687 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:00:30.153286 18687 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:00:30.153422 18687 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:00:30.160403 18687 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:00:30.160522 18687 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:00:30.160673 18687 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260924 10:00:30.170023 18687 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790243735634835-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790243735634835-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790243735634835-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790243735634835-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45923
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790243735634835-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:00:30.171314 18687 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:00:30.172576 18687 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:00:30.184772 18692 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:00:30.184736 18693 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:00:30.184799 18695 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:00:30.185952 18687 server_base.cc:1034] running on GCE node
I20260924 10:00:30.186520 18687 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:00:30.187423 18687 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:00:30.188601 18687 hybrid_clock.cc:648] HybridClock initialized: now 1790244030188549 us; error 51 us; skew 500 ppm
I20260924 10:00:30.188881 18687 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:00:30 dist-test-slave-w8d2 krb5kdc[18581](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244030, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:00:30.191834 18687 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260924 10:00:30.193148 18687 webserver.cc:460] Webserver started at http://127.0.97.1:34179/ using document root <none> and password file <none>
I20260924 10:00:30.193782 18687 fs_manager.cc:359] Metadata directory not provided
I20260924 10:00:30.193876 18687 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:00:30.194114 18687 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:00:30.196907 18687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790243735634835-388-0/minicluster-data/ts-0/data/instance:
uuid: "217beb0b90284b199f5b2959b8de96d8"
format_stamp: "Formatted at 2026-09-24 10:00:30 on dist-test-slave-w8d2"
server_key: "13a81eacb83d35b3dcdd5bef1abd61b8"
server_key_iv: "40590544f37c954e67f730f2726880c1"
server_key_version: "encryptionkey@0"
I20260924 10:00:30.197608 18687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790243735634835-388-0/minicluster-data/ts-0/wal/instance:
uuid: "217beb0b90284b199f5b2959b8de96d8"
format_stamp: "Formatted at 2026-09-24 10:00:30 on dist-test-slave-w8d2"
server_key: "13a81eacb83d35b3dcdd5bef1abd61b8"
server_key_iv: "40590544f37c954e67f730f2726880c1"
server_key_version: "encryptionkey@0"
I20260924 10:00:30.202306 18687 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260924 10:00:30.205679 18702 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:00:30.206977 18687 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260924 10:00:30.207111 18687 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790243735634835-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790243735634835-388-0/minicluster-data/ts-0/wal
uuid: "217beb0b90284b199f5b2959b8de96d8"
format_stamp: "Formatted at 2026-09-24 10:00:30 on dist-test-slave-w8d2"
server_key: "13a81eacb83d35b3dcdd5bef1abd61b8"
server_key_iv: "40590544f37c954e67f730f2726880c1"
server_key_version: "encryptionkey@0"
I20260924 10:00:30.207334 18687 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790243735634835-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790243735634835-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790243735634835-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:00:30.226473 18687 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:00:30.230526 18687 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:00:30.230782 18687 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:00:30.231745 18687 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:00:30.232980 18687 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:00:30.233058 18687 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:00:30.233173 18687 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:00:30.233222 18687 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:00:30.261305 18687 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:41027
I20260924 10:00:30.261346 18814 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:41027 every 8 connection(s)
I20260924 10:00:30.262492 18687 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790243735634835-388-0/minicluster-data/ts-0/data/info.pb
I20260924 10:00:30.270952 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu as pid 18687
I20260924 10:00:30.271112 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790243735634835-388-0/minicluster-data/ts-0/wal/instance
I20260924 10:00:30.271524 388 external_mini_cluster.cc:1428] Setting key 3982348692171f99f6f771c530974b92
Sep 24 10:00:30 dist-test-slave-w8d2 krb5kdc[18581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244030, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 10:00:30.281539 18817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:30.264830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:52851 (local address 127.0.97.62:45923)
0924 10:00:30.265237 (+ 407us) server_negotiation.cc:206] Beginning negotiation
0924 10:00:30.265241 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:00:30.266084 (+ 843us) server_negotiation.cc:407] Connection header received
0924 10:00:30.267891 (+ 1807us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:00:30.267896 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:00:30.267977 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:00:30.268070 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:00:30.269508 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:00:30.270214 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:00:30.271838 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:00:30.272232 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:00:30.272338 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:00:30.276238 (+ 3900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:00:30.276243 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:00:30.276246 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:00:30.276276 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:00:30.278681 (+ 2405us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:00:30.279339 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:00:30.279344 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:00:30.279348 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:00:30.279484 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:00:30.279880 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:00:30.279884 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:00:30.279887 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:00:30.280217 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:00:30.280359 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0924 10:00:30.280950 (+ 591us) server_negotiation.cc:299] Negotiation successful
0924 10:00:30.281171 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":210,"threads_started":1}
I20260924 10:00:30.282212 18818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:30.265260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45923 (local address 127.0.97.1:52851)
0924 10:00:30.265917 (+ 657us) negotiation.cc:107] Waiting for socket to connect
0924 10:00:30.265960 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 10:00:30.267634 (+ 1674us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:00:30.268311 (+ 677us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:00:30.268320 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:00:30.268798 (+ 478us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:00:30.269339 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:00:30.269355 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:00:30.270355 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:00:30.270359 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:00:30.271705 (+ 1346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:00:30.271717 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:00:30.272438 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:00:30.272446 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:00:30.272670 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:00:30.273869 (+ 1199us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:00:30.273894 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:00:30.276029 (+ 2135us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:00:30.278802 (+ 2773us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:00:30.278811 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:00:30.278827 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:00:30.279213 (+ 386us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:00:30.279603 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:00:30.279607 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:00:30.279609 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:00:30.279783 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:00:30.280318 (+ 535us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:00:30.280323 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:00:30.280620 (+ 297us) client_negotiation.cc:769] Sending connection context
0924 10:00:30.280827 (+ 207us) client_negotiation.cc:240] Negotiation successful
0924 10:00:30.281185 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":254,"threads_started":1}
I20260924 10:00:30.283545 18815 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45923
I20260924 10:00:30.283806 18815 heartbeater.cc:464] Registering TS with master...
I20260924 10:00:30.284446 18815 heartbeater.cc:511] Master 127.0.97.62:45923 requested a full tablet report, sending...
I20260924 10:00:30.286278 18626 ts_manager.cc:194] Registered new tserver with Master: 217beb0b90284b199f5b2959b8de96d8 (127.0.97.1:41027)
I20260924 10:00:30.287688 18626 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:52851
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790243735634835-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790243735634835-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260924 10:00:30.319767 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
/tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790243735634835-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790243735634835-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790243735634835-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790243735634835-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45923
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790243735634835-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790243735634835-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790243735634835-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790243735634835-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:00:30.626788 18821 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:00:30.627074 18821 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:00:30.627177 18821 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:00:30.634701 18821 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:00:30.634809 18821 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:00:30.634934 18821 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260924 10:00:30.644599 18821 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790243735634835-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790243735634835-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790243735634835-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790243735634835-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45923
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790243735634835-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:00:30.645825 18821 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:00:30.647034 18821 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:00:30.659416 18827 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:00:30.659514 18829 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:00:30.659746 18826 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:00:30.660777 18821 server_base.cc:1034] running on GCE node
I20260924 10:00:30.661319 18821 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:00:30.662179 18821 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:00:30.663326 18821 hybrid_clock.cc:648] HybridClock initialized: now 1790244030663286 us; error 30 us; skew 500 ppm
I20260924 10:00:30.663596 18821 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:00:30 dist-test-slave-w8d2 krb5kdc[18581](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244030, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:00:30.666324 18821 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260924 10:00:30.667696 18821 webserver.cc:460] Webserver started at http://127.0.97.2:35905/ using document root <none> and password file <none>
I20260924 10:00:30.668300 18821 fs_manager.cc:359] Metadata directory not provided
I20260924 10:00:30.668392 18821 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:00:30.668669 18821 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:00:30.671420 18821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790243735634835-388-0/minicluster-data/ts-1/data/instance:
uuid: "c971e95cdaec47a18163abd42a952013"
format_stamp: "Formatted at 2026-09-24 10:00:30 on dist-test-slave-w8d2"
server_key: "51d0608fea33b4c5ab47c666a2525472"
server_key_iv: "62ec829fe5cec7c6053ed04446e2e4c9"
server_key_version: "encryptionkey@0"
I20260924 10:00:30.672101 18821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790243735634835-388-0/minicluster-data/ts-1/wal/instance:
uuid: "c971e95cdaec47a18163abd42a952013"
format_stamp: "Formatted at 2026-09-24 10:00:30 on dist-test-slave-w8d2"
server_key: "51d0608fea33b4c5ab47c666a2525472"
server_key_iv: "62ec829fe5cec7c6053ed04446e2e4c9"
server_key_version: "encryptionkey@0"
I20260924 10:00:30.676733 18821 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 10:00:30.679893 18836 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:00:30.681098 18821 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 10:00:30.681216 18821 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790243735634835-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790243735634835-388-0/minicluster-data/ts-1/wal
uuid: "c971e95cdaec47a18163abd42a952013"
format_stamp: "Formatted at 2026-09-24 10:00:30 on dist-test-slave-w8d2"
server_key: "51d0608fea33b4c5ab47c666a2525472"
server_key_iv: "62ec829fe5cec7c6053ed04446e2e4c9"
server_key_version: "encryptionkey@0"
I20260924 10:00:30.681377 18821 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790243735634835-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790243735634835-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790243735634835-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:00:30.696177 18821 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:00:30.700089 18821 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:00:30.700361 18821 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:00:30.701225 18821 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:00:30.702435 18821 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:00:30.702500 18821 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:00:30.702602 18821 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:00:30.702651 18821 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:00:30.731950 18821 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:42301
I20260924 10:00:30.731987 18948 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:42301 every 8 connection(s)
I20260924 10:00:30.733165 18821 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790243735634835-388-0/minicluster-data/ts-1/data/info.pb
I20260924 10:00:30.740005 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu as pid 18821
I20260924 10:00:30.740191 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790243735634835-388-0/minicluster-data/ts-1/wal/instance
I20260924 10:00:30.740777 388 external_mini_cluster.cc:1428] Setting key 7bfa4aa5c0199eef816dec4c88787e58
Sep 24 10:00:30 dist-test-slave-w8d2 krb5kdc[18581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244030, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 10:00:30.755053 18817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:30.735631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:46227 (local address 127.0.97.62:45923)
0924 10:00:30.735807 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0924 10:00:30.735812 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:00:30.736880 (+ 1068us) server_negotiation.cc:407] Connection header received
0924 10:00:30.738677 (+ 1797us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:00:30.738683 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:00:30.738761 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:00:30.738846 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:00:30.740564 (+ 1718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:00:30.741690 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:00:30.744080 (+ 2390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:00:30.744678 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:00:30.744796 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:00:30.749113 (+ 4317us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:00:30.749121 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:00:30.749126 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:00:30.749200 (+ 74us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:00:30.752127 (+ 2927us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:00:30.752873 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:00:30.752877 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:00:30.752880 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:00:30.753009 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:00:30.753417 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:00:30.753421 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:00:30.753423 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:00:30.753678 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:00:30.753804 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0924 10:00:30.754480 (+ 676us) server_negotiation.cc:299] Negotiation successful
0924 10:00:30.754711 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 10:00:30.756228 18951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:30.736071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45923 (local address 127.0.97.2:46227)
0924 10:00:30.736732 (+ 661us) negotiation.cc:107] Waiting for socket to connect
0924 10:00:30.736773 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 10:00:30.738418 (+ 1645us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:00:30.739076 (+ 658us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:00:30.739085 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:00:30.739607 (+ 522us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:00:30.740387 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:00:30.740414 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:00:30.741911 (+ 1497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:00:30.741917 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:00:30.743919 (+ 2002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:00:30.743935 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:00:30.744930 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:00:30.744938 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:00:30.745207 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:00:30.745985 (+ 778us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:00:30.746016 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:00:30.748776 (+ 2760us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:00:30.752269 (+ 3493us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:00:30.752278 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:00:30.752296 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:00:30.752742 (+ 446us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:00:30.753111 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:00:30.753116 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:00:30.753120 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:00:30.753301 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:00:30.753793 (+ 492us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:00:30.753802 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:00:30.754131 (+ 329us) client_negotiation.cc:769] Sending connection context
0924 10:00:30.754374 (+ 243us) client_negotiation.cc:240] Negotiation successful
0924 10:00:30.754897 (+ 523us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":255,"threads_started":1}
I20260924 10:00:30.757687 18949 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45923
I20260924 10:00:30.757958 18949 heartbeater.cc:464] Registering TS with master...
I20260924 10:00:30.758633 18949 heartbeater.cc:511] Master 127.0.97.62:45923 requested a full tablet report, sending...
I20260924 10:00:30.760063 18626 ts_manager.cc:194] Registered new tserver with Master: c971e95cdaec47a18163abd42a952013 (127.0.97.2:42301)
I20260924 10:00:30.761155 18626 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:46227
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790243735634835-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790243735634835-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260924 10:00:30.792131 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
/tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790243735634835-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790243735634835-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790243735634835-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790243735634835-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45923
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790243735634835-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790243735634835-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790243735634835-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790243735634835-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:00:31.098482 18954 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:00:31.098771 18954 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:00:31.098863 18954 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:00:31.106092 18954 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:00:31.106204 18954 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:00:31.106320 18954 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260924 10:00:31.115952 18954 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790243735634835-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790243735634835-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790243735634835-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790243735634835-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45923
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790243735634835-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:00:31.117130 18954 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:00:31.118350 18954 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:00:31.130203 18962 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:00:31.130208 18960 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:00:31.130576 18959 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:00:31.131033 18954 server_base.cc:1034] running on GCE node
I20260924 10:00:31.131625 18954 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:00:31.132453 18954 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:00:31.133642 18954 hybrid_clock.cc:648] HybridClock initialized: now 1790244031133597 us; error 36 us; skew 500 ppm
I20260924 10:00:31.133921 18954 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:00:31 dist-test-slave-w8d2 krb5kdc[18581](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244031, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:00:31.143944 18954 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260924 10:00:31.145316 18954 webserver.cc:460] Webserver started at http://127.0.97.3:40001/ using document root <none> and password file <none>
I20260924 10:00:31.145946 18954 fs_manager.cc:359] Metadata directory not provided
I20260924 10:00:31.146047 18954 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:00:31.146291 18954 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:00:31.149076 18954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790243735634835-388-0/minicluster-data/ts-2/data/instance:
uuid: "5902a70ef84d41f3887928dea571100d"
format_stamp: "Formatted at 2026-09-24 10:00:31 on dist-test-slave-w8d2"
server_key: "fa874b3dd74abd231a5976baee952094"
server_key_iv: "f54f02a7d1364ac29c6c58802c505138"
server_key_version: "encryptionkey@0"
I20260924 10:00:31.149883 18954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790243735634835-388-0/minicluster-data/ts-2/wal/instance:
uuid: "5902a70ef84d41f3887928dea571100d"
format_stamp: "Formatted at 2026-09-24 10:00:31 on dist-test-slave-w8d2"
server_key: "fa874b3dd74abd231a5976baee952094"
server_key_iv: "f54f02a7d1364ac29c6c58802c505138"
server_key_version: "encryptionkey@0"
I20260924 10:00:31.154686 18954 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260924 10:00:31.157979 18969 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:00:31.159209 18954 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 10:00:31.159381 18954 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790243735634835-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790243735634835-388-0/minicluster-data/ts-2/wal
uuid: "5902a70ef84d41f3887928dea571100d"
format_stamp: "Formatted at 2026-09-24 10:00:31 on dist-test-slave-w8d2"
server_key: "fa874b3dd74abd231a5976baee952094"
server_key_iv: "f54f02a7d1364ac29c6c58802c505138"
server_key_version: "encryptionkey@0"
I20260924 10:00:31.159559 18954 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790243735634835-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790243735634835-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790243735634835-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:00:31.174153 18954 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:00:31.178063 18954 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:00:31.178306 18954 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:00:31.179170 18954 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:00:31.180413 18954 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:00:31.180485 18954 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:00:31.180596 18954 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:00:31.180647 18954 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:00:31.208436 18954 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:45681
I20260924 10:00:31.208467 19081 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:45681 every 8 connection(s)
I20260924 10:00:31.209606 18954 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790243735634835-388-0/minicluster-data/ts-2/data/info.pb
I20260924 10:00:31.212859 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu as pid 18954
I20260924 10:00:31.213018 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790243735634835-388-0/minicluster-data/ts-2/wal/instance
I20260924 10:00:31.213478 388 external_mini_cluster.cc:1428] Setting key d0ad6117fd60970930735c90c4bf0abe
Sep 24 10:00:31 dist-test-slave-w8d2 krb5kdc[18581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244031, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 10:00:31.228386 18817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:31.211956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:38217 (local address 127.0.97.62:45923)
0924 10:00:31.212128 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0924 10:00:31.212133 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:00:31.213385 (+ 1252us) server_negotiation.cc:407] Connection header received
0924 10:00:31.215454 (+ 2069us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:00:31.215459 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:00:31.215539 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:00:31.215622 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:00:31.217118 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:00:31.217820 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:00:31.219289 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:00:31.219680 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:00:31.219784 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:00:31.223052 (+ 3268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:00:31.223057 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:00:31.223060 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:00:31.223091 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:00:31.225519 (+ 2428us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:00:31.226178 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:00:31.226212 (+ 34us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:00:31.226215 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:00:31.226324 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:00:31.226787 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:00:31.226792 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:00:31.226795 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:00:31.227133 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:00:31.227272 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0924 10:00:31.227937 (+ 665us) server_negotiation.cc:299] Negotiation successful
0924 10:00:31.228149 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260924 10:00:31.229318 19084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:31.212460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45923 (local address 127.0.97.3:38217)
0924 10:00:31.213212 (+ 752us) negotiation.cc:107] Waiting for socket to connect
0924 10:00:31.213255 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 10:00:31.215171 (+ 1916us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:00:31.215891 (+ 720us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:00:31.215903 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:00:31.216425 (+ 522us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:00:31.216964 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:00:31.216980 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:00:31.217960 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:00:31.217964 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:00:31.219137 (+ 1173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:00:31.219148 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:00:31.219882 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:00:31.219889 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:00:31.220106 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:00:31.220690 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:00:31.220715 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:00:31.222843 (+ 2128us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:00:31.225640 (+ 2797us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:00:31.225648 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:00:31.225664 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:00:31.226077 (+ 413us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:00:31.226431 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:00:31.226435 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:00:31.226437 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:00:31.226663 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:00:31.227305 (+ 642us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:00:31.227313 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:00:31.227633 (+ 320us) client_negotiation.cc:769] Sending connection context
0924 10:00:31.227855 (+ 222us) client_negotiation.cc:240] Negotiation successful
0924 10:00:31.228261 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":310,"threads_started":1}
I20260924 10:00:31.230500 19082 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45923
I20260924 10:00:31.230800 19082 heartbeater.cc:464] Registering TS with master...
I20260924 10:00:31.231562 19082 heartbeater.cc:511] Master 127.0.97.62:45923 requested a full tablet report, sending...
I20260924 10:00:31.232863 18626 ts_manager.cc:194] Registered new tserver with Master: 5902a70ef84d41f3887928dea571100d (127.0.97.3:45681)
I20260924 10:00:31.233629 18626 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:38217
I20260924 10:00:31.239928 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 10:00:31.256541 18817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:31.243715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54296 (local address 127.0.97.62:45923)
0924 10:00:31.243869 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0924 10:00:31.243873 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:00:31.244292 (+ 419us) server_negotiation.cc:407] Connection header received
0924 10:00:31.244418 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:00:31.244421 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:00:31.244477 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:00:31.244551 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:00:31.245906 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:00:31.246604 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:00:31.248091 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:00:31.248540 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:00:31.248644 (+ 104us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:00:31.251417 (+ 2773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:00:31.251423 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:00:31.251426 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:00:31.251460 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:00:31.253909 (+ 2449us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:00:31.254706 (+ 797us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:00:31.254710 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:00:31.254712 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:00:31.254828 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:00:31.255210 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:00:31.255213 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:00:31.255215 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:00:31.255554 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:00:31.255738 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0924 10:00:31.256173 (+ 435us) server_negotiation.cc:299] Negotiation successful
0924 10:00:31.256318 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 10:00:31.260324 18626 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54296:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 10:00:31.267194 18626 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 10:00:31.289309 19096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:31.279816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41027 (local address 127.0.0.1:45624)
0924 10:00:31.280246 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0924 10:00:31.280262 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 10:00:31.280362 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:00:31.281536 (+ 1174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:00:31.281540 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:00:31.281563 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:00:31.281946 (+ 383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:00:31.281959 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:00:31.283836 (+ 1877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:00:31.283841 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:00:31.286360 (+ 2519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:00:31.286375 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:00:31.288401 (+ 2026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:00:31.288408 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:00:31.288593 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:00:31.288643 (+ 50us) client_negotiation.cc:769] Sending connection context
0924 10:00:31.288767 (+ 124us) client_negotiation.cc:240] Negotiation successful
0924 10:00:31.288960 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":592,"threads_started":1}
I20260924 10:00:31.290189 19094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:31.278530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:42301 (local address 127.0.0.1:56706)
0924 10:00:31.279505 (+ 975us) negotiation.cc:107] Waiting for socket to connect
0924 10:00:31.279546 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 10:00:31.279724 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:00:31.280974 (+ 1250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:00:31.280981 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:00:31.281010 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:00:31.281454 (+ 444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:00:31.281469 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:00:31.284611 (+ 3142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:00:31.284617 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:00:31.286656 (+ 2039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:00:31.286669 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:00:31.289503 (+ 2834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:00:31.289510 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:00:31.289676 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:00:31.289707 (+ 31us) client_negotiation.cc:769] Sending connection context
0924 10:00:31.289759 (+ 52us) client_negotiation.cc:240] Negotiation successful
0924 10:00:31.289873 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":677,"thread_start_us":547,"threads_started":1}
I20260924 10:00:31.290279 19097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:31.280279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45624 (local address 127.0.97.1:41027)
0924 10:00:31.280804 (+ 525us) server_negotiation.cc:206] Beginning negotiation
0924 10:00:31.280812 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 10:00:31.280839 (+ 27us) server_negotiation.cc:407] Connection header received
0924 10:00:31.280939 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:00:31.280946 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:00:31.281150 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:00:31.281327 (+ 177us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:00:31.282123 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:00:31.283673 (+ 1550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:00:31.286531 (+ 2858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:00:31.287947 (+ 1416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:00:31.288060 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:00:31.288272 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0924 10:00:31.289826 (+ 1554us) server_negotiation.cc:299] Negotiation successful
0924 10:00:31.289993 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":373,"thread_start_us":483,"threads_started":1}
I20260924 10:00:31.290599 18815 heartbeater.cc:503] Master 127.0.97.62:45923 was elected leader, sending a full tablet report...
I20260924 10:00:31.291487 19095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:31.278689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56706 (local address 127.0.97.2:42301)
0924 10:00:31.280343 (+ 1654us) server_negotiation.cc:206] Beginning negotiation
0924 10:00:31.280350 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:00:31.280371 (+ 21us) server_negotiation.cc:407] Connection header received
0924 10:00:31.280479 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:00:31.280487 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:00:31.280743 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:00:31.281645 (+ 902us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:00:31.281839 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:00:31.284468 (+ 2629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:00:31.287919 (+ 3451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:00:31.289277 (+ 1358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:00:31.290399 (+ 1122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:00:31.290582 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0924 10:00:31.290655 (+ 73us) server_negotiation.cc:299] Negotiation successful
0924 10:00:31.290804 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1532,"thread_start_us":1418,"threads_started":1}
I20260924 10:00:31.295552 18884 tablet_service.cc:1467] Processing CreateTablet for tablet 26b6f8c945c541f09657d90d3edadb6e (DEFAULT_TABLE table=test-table [id=1bb20ae5fabc4665be354f212bbe7147]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:00:31.295912 18750 tablet_service.cc:1467] Processing CreateTablet for tablet 26b6f8c945c541f09657d90d3edadb6e (DEFAULT_TABLE table=test-table [id=1bb20ae5fabc4665be354f212bbe7147]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:00:31.297014 18884 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26b6f8c945c541f09657d90d3edadb6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:00:31.297240 18750 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26b6f8c945c541f09657d90d3edadb6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:00:31.298082 19096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:31.280668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:45681 (local address 127.0.0.1:39294)
0924 10:00:31.289561 (+ 8893us) negotiation.cc:107] Waiting for socket to connect
0924 10:00:31.289583 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 10:00:31.289669 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:00:31.291950 (+ 2281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:00:31.291955 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:00:31.291980 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:00:31.292139 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:00:31.292149 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:00:31.294059 (+ 1910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:00:31.294064 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:00:31.295777 (+ 1713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:00:31.295792 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:00:31.297433 (+ 1641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:00:31.297439 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:00:31.297617 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:00:31.297649 (+ 32us) client_negotiation.cc:769] Sending connection context
0924 10:00:31.297715 (+ 66us) client_negotiation.cc:240] Negotiation successful
0924 10:00:31.297840 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8793,"thread_start_us":10989,"threads_started":1}
I20260924 10:00:31.299047 19099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:31.280977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39294 (local address 127.0.97.3:45681)
0924 10:00:31.291219 (+ 10242us) server_negotiation.cc:206] Beginning negotiation
0924 10:00:31.291258 (+ 39us) server_negotiation.cc:399] Waiting for connection header
0924 10:00:31.291293 (+ 35us) server_negotiation.cc:407] Connection header received
0924 10:00:31.291386 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:00:31.291394 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:00:31.291729 (+ 335us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:00:31.292299 (+ 570us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:00:31.292542 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:00:31.293922 (+ 1380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:00:31.295944 (+ 2022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:00:31.297246 (+ 1302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:00:31.298309 (+ 1063us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:00:31.298534 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0924 10:00:31.298601 (+ 67us) server_negotiation.cc:299] Negotiation successful
0924 10:00:31.298741 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10069,"thread_start_us":9885,"threads_started":1}
I20260924 10:00:31.302580 19017 tablet_service.cc:1467] Processing CreateTablet for tablet 26b6f8c945c541f09657d90d3edadb6e (DEFAULT_TABLE table=test-table [id=1bb20ae5fabc4665be354f212bbe7147]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:00:31.303915 19017 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26b6f8c945c541f09657d90d3edadb6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:00:31.308194 19100 tablet_bootstrap.cc:492] T 26b6f8c945c541f09657d90d3edadb6e P 217beb0b90284b199f5b2959b8de96d8: Bootstrap starting.
I20260924 10:00:31.311749 19100 tablet_bootstrap.cc:654] T 26b6f8c945c541f09657d90d3edadb6e P 217beb0b90284b199f5b2959b8de96d8: Neither blocks nor log segments found. Creating new log.
I20260924 10:00:31.312004 19101 tablet_bootstrap.cc:492] T 26b6f8c945c541f09657d90d3edadb6e P c971e95cdaec47a18163abd42a952013: Bootstrap starting.
I20260924 10:00:31.313133 19100 log.cc:824] T 26b6f8c945c541f09657d90d3edadb6e P 217beb0b90284b199f5b2959b8de96d8: Log is configured to *not* fsync() on all Append() calls
I20260924 10:00:31.315171 19101 tablet_bootstrap.cc:654] T 26b6f8c945c541f09657d90d3edadb6e P c971e95cdaec47a18163abd42a952013: Neither blocks nor log segments found. Creating new log.
I20260924 10:00:31.316186 19100 tablet_bootstrap.cc:492] T 26b6f8c945c541f09657d90d3edadb6e P 217beb0b90284b199f5b2959b8de96d8: No bootstrap required, opened a new log
I20260924 10:00:31.316347 19101 log.cc:824] T 26b6f8c945c541f09657d90d3edadb6e P c971e95cdaec47a18163abd42a952013: Log is configured to *not* fsync() on all Append() calls
I20260924 10:00:31.316459 19100 ts_tablet_manager.cc:1397] T 26b6f8c945c541f09657d90d3edadb6e P 217beb0b90284b199f5b2959b8de96d8: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260924 10:00:31.317003 19103 tablet_bootstrap.cc:492] T 26b6f8c945c541f09657d90d3edadb6e P 5902a70ef84d41f3887928dea571100d: Bootstrap starting.
I20260924 10:00:31.320001 19103 tablet_bootstrap.cc:654] T 26b6f8c945c541f09657d90d3edadb6e P 5902a70ef84d41f3887928dea571100d: Neither blocks nor log segments found. Creating new log.
I20260924 10:00:31.320779 19101 tablet_bootstrap.cc:492] T 26b6f8c945c541f09657d90d3edadb6e P c971e95cdaec47a18163abd42a952013: No bootstrap required, opened a new log
I20260924 10:00:31.320886 19103 log.cc:824] T 26b6f8c945c541f09657d90d3edadb6e P 5902a70ef84d41f3887928dea571100d: Log is configured to *not* fsync() on all Append() calls
I20260924 10:00:31.321048 19101 ts_tablet_manager.cc:1397] T 26b6f8c945c541f09657d90d3edadb6e P c971e95cdaec47a18163abd42a952013: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.006s
I20260924 10:00:31.323068 19103 tablet_bootstrap.cc:492] T 26b6f8c945c541f09657d90d3edadb6e P 5902a70ef84d41f3887928dea571100d: No bootstrap required, opened a new log
I20260924 10:00:31.323314 19103 ts_tablet_manager.cc:1397] T 26b6f8c945c541f09657d90d3edadb6e P 5902a70ef84d41f3887928dea571100d: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260924 10:00:31.326385 19100 raft_consensus.cc:348] T 26b6f8c945c541f09657d90d3edadb6e P 217beb0b90284b199f5b2959b8de96d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c971e95cdaec47a18163abd42a952013" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42301 } } peers { permanent_uuid: "217beb0b90284b199f5b2959b8de96d8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41027 } } peers { permanent_uuid: "5902a70ef84d41f3887928dea571100d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45681 } }
I20260924 10:00:31.326809 19100 raft_consensus.cc:374] T 26b6f8c945c541f09657d90d3edadb6e P 217beb0b90284b199f5b2959b8de96d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:00:31.326903 19100 raft_consensus.cc:729] T 26b6f8c945c541f09657d90d3edadb6e P 217beb0b90284b199f5b2959b8de96d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 217beb0b90284b199f5b2959b8de96d8, State: Initialized, Role: FOLLOWER
I20260924 10:00:31.327538 19100 consensus_queue.cc:260] T 26b6f8c945c541f09657d90d3edadb6e P 217beb0b90284b199f5b2959b8de96d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c971e95cdaec47a18163abd42a952013" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42301 } } peers { permanent_uuid: "217beb0b90284b199f5b2959b8de96d8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41027 } } peers { permanent_uuid: "5902a70ef84d41f3887928dea571100d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45681 } }
I20260924 10:00:31.329205 19100 ts_tablet_manager.cc:1428] T 26b6f8c945c541f09657d90d3edadb6e P 217beb0b90284b199f5b2959b8de96d8: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20260924 10:00:31.331421 19101 raft_consensus.cc:348] T 26b6f8c945c541f09657d90d3edadb6e P c971e95cdaec47a18163abd42a952013 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c971e95cdaec47a18163abd42a952013" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42301 } } peers { permanent_uuid: "217beb0b90284b199f5b2959b8de96d8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41027 } } peers { permanent_uuid: "5902a70ef84d41f3887928dea571100d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45681 } }
I20260924 10:00:31.331847 19101 raft_consensus.cc:374] T 26b6f8c945c541f09657d90d3edadb6e P c971e95cdaec47a18163abd42a952013 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:00:31.331954 19101 raft_consensus.cc:729] T 26b6f8c945c541f09657d90d3edadb6e P c971e95cdaec47a18163abd42a952013 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c971e95cdaec47a18163abd42a952013, State: Initialized, Role: FOLLOWER
I20260924 10:00:31.332520 19101 consensus_queue.cc:260] T 26b6f8c945c541f09657d90d3edadb6e P c971e95cdaec47a18163abd42a952013 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c971e95cdaec47a18163abd42a952013" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42301 } } peers { permanent_uuid: "217beb0b90284b199f5b2959b8de96d8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41027 } } peers { permanent_uuid: "5902a70ef84d41f3887928dea571100d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45681 } }
I20260924 10:00:31.333031 19103 raft_consensus.cc:348] T 26b6f8c945c541f09657d90d3edadb6e P 5902a70ef84d41f3887928dea571100d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c971e95cdaec47a18163abd42a952013" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42301 } } peers { permanent_uuid: "217beb0b90284b199f5b2959b8de96d8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41027 } } peers { permanent_uuid: "5902a70ef84d41f3887928dea571100d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45681 } }
I20260924 10:00:31.333433 19103 raft_consensus.cc:374] T 26b6f8c945c541f09657d90d3edadb6e P 5902a70ef84d41f3887928dea571100d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:00:31.333520 19103 raft_consensus.cc:729] T 26b6f8c945c541f09657d90d3edadb6e P 5902a70ef84d41f3887928dea571100d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5902a70ef84d41f3887928dea571100d, State: Initialized, Role: FOLLOWER
I20260924 10:00:31.333598 18949 heartbeater.cc:503] Master 127.0.97.62:45923 was elected leader, sending a full tablet report...
I20260924 10:00:31.334056 19103 consensus_queue.cc:260] T 26b6f8c945c541f09657d90d3edadb6e P 5902a70ef84d41f3887928dea571100d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c971e95cdaec47a18163abd42a952013" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42301 } } peers { permanent_uuid: "217beb0b90284b199f5b2959b8de96d8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41027 } } peers { permanent_uuid: "5902a70ef84d41f3887928dea571100d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45681 } }
I20260924 10:00:31.334273 19101 ts_tablet_manager.cc:1428] T 26b6f8c945c541f09657d90d3edadb6e P c971e95cdaec47a18163abd42a952013: Time spent starting tablet: real 0.013s user 0.014s sys 0.000s
I20260924 10:00:31.335119 19082 heartbeater.cc:503] Master 127.0.97.62:45923 was elected leader, sending a full tablet report...
I20260924 10:00:31.335819 19103 ts_tablet_manager.cc:1428] T 26b6f8c945c541f09657d90d3edadb6e P 5902a70ef84d41f3887928dea571100d: Time spent starting tablet: real 0.012s user 0.007s sys 0.004s
I20260924 10:00:31.452580 19108 raft_consensus.cc:482] T 26b6f8c945c541f09657d90d3edadb6e P 5902a70ef84d41f3887928dea571100d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 10:00:31.452857 19108 raft_consensus.cc:504] T 26b6f8c945c541f09657d90d3edadb6e P 5902a70ef84d41f3887928dea571100d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c971e95cdaec47a18163abd42a952013" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42301 } } peers { permanent_uuid: "217beb0b90284b199f5b2959b8de96d8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41027 } } peers { permanent_uuid: "5902a70ef84d41f3887928dea571100d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45681 } }
I20260924 10:00:31.454463 19108 leader_election.cc:288] T 26b6f8c945c541f09657d90d3edadb6e P 5902a70ef84d41f3887928dea571100d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c971e95cdaec47a18163abd42a952013 (127.0.97.2:42301), 217beb0b90284b199f5b2959b8de96d8 (127.0.97.1:41027)
I20260924 10:00:31.460820 19095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:31.454831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:54939 (local address 127.0.97.2:42301)
0924 10:00:31.455059 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0924 10:00:31.455065 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:00:31.455092 (+ 27us) server_negotiation.cc:407] Connection header received
0924 10:00:31.455350 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:00:31.455355 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:00:31.455441 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:00:31.455539 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:00:31.456071 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:00:31.457010 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:00:31.458855 (+ 1845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:00:31.460031 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:00:31.460126 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:00:31.460193 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0924 10:00:31.460551 (+ 358us) server_negotiation.cc:299] Negotiation successful
0924 10:00:31.460637 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 10:00:31.460970 19084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:31.454834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:42301 (local address 127.0.97.3:54939)
0924 10:00:31.455023 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0924 10:00:31.455041 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 10:00:31.455164 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:00:31.455714 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:00:31.455722 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:00:31.455748 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:00:31.455908 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:00:31.455915 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:00:31.457126 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:00:31.457132 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:00:31.458703 (+ 1571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:00:31.458715 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:00:31.460185 (+ 1470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:00:31.460191 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:00:31.460405 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:00:31.460441 (+ 36us) client_negotiation.cc:769] Sending connection context
0924 10:00:31.460536 (+ 95us) client_negotiation.cc:240] Negotiation successful
0924 10:00:31.460664 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54,"mutex_wait_us":19}
W20260924 10:00:31.461139 19083 tablet.cc:2369] T 26b6f8c945c541f09657d90d3edadb6e P 5902a70ef84d41f3887928dea571100d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 10:00:31.462019 19097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:31.455481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:59639 (local address 127.0.97.1:41027)
0924 10:00:31.455668 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0924 10:00:31.455674 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:00:31.455990 (+ 316us) server_negotiation.cc:407] Connection header received
0924 10:00:31.456116 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:00:31.456120 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:00:31.456209 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:00:31.456299 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:00:31.456830 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:00:31.457795 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:00:31.459998 (+ 2203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:00:31.461138 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:00:31.461249 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:00:31.461341 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0924 10:00:31.461668 (+ 327us) server_negotiation.cc:299] Negotiation successful
0924 10:00:31.461788 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260924 10:00:31.462093 19109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:31.455211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41027 (local address 127.0.97.3:59639)
0924 10:00:31.455892 (+ 681us) negotiation.cc:107] Waiting for socket to connect
0924 10:00:31.455905 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 10:00:31.456005 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:00:31.456324 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:00:31.456329 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:00:31.456353 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:00:31.456684 (+ 331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:00:31.456694 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:00:31.457945 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:00:31.457948 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:00:31.459838 (+ 1890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:00:31.459853 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:00:31.461410 (+ 1557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:00:31.461415 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:00:31.461531 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:00:31.461553 (+ 22us) client_negotiation.cc:769] Sending connection context
0924 10:00:31.461627 (+ 74us) client_negotiation.cc:240] Negotiation successful
0924 10:00:31.461739 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":586,"mutex_wait_us":20,"thread_start_us":359,"threads_started":1}
I20260924 10:00:31.462110 18904 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26b6f8c945c541f09657d90d3edadb6e" candidate_uuid: "5902a70ef84d41f3887928dea571100d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c971e95cdaec47a18163abd42a952013" is_pre_election: true
I20260924 10:00:31.462599 18904 raft_consensus.cc:2454] T 26b6f8c945c541f09657d90d3edadb6e P c971e95cdaec47a18163abd42a952013 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5902a70ef84d41f3887928dea571100d in term 0.
I20260924 10:00:31.463006 18770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26b6f8c945c541f09657d90d3edadb6e" candidate_uuid: "5902a70ef84d41f3887928dea571100d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "217beb0b90284b199f5b2959b8de96d8" is_pre_election: true
I20260924 10:00:31.463245 18973 leader_election.cc:302] T 26b6f8c945c541f09657d90d3edadb6e P 5902a70ef84d41f3887928dea571100d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5902a70ef84d41f3887928dea571100d, c971e95cdaec47a18163abd42a952013; no voters:
I20260924 10:00:31.463613 19108 raft_consensus.cc:2781] T 26b6f8c945c541f09657d90d3edadb6e P 5902a70ef84d41f3887928dea571100d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 10:00:31.463572 18770 raft_consensus.cc:2454] T 26b6f8c945c541f09657d90d3edadb6e P 217beb0b90284b199f5b2959b8de96d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5902a70ef84d41f3887928dea571100d in term 0.
I20260924 10:00:31.463734 19108 raft_consensus.cc:482] T 26b6f8c945c541f09657d90d3edadb6e P 5902a70ef84d41f3887928dea571100d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 10:00:31.463825 19108 raft_consensus.cc:3037] T 26b6f8c945c541f09657d90d3edadb6e P 5902a70ef84d41f3887928dea571100d [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:00:31.465936 19108 raft_consensus.cc:504] T 26b6f8c945c541f09657d90d3edadb6e P 5902a70ef84d41f3887928dea571100d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c971e95cdaec47a18163abd42a952013" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42301 } } peers { permanent_uuid: "217beb0b90284b199f5b2959b8de96d8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41027 } } peers { permanent_uuid: "5902a70ef84d41f3887928dea571100d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45681 } }
I20260924 10:00:31.466459 19108 leader_election.cc:288] T 26b6f8c945c541f09657d90d3edadb6e P 5902a70ef84d41f3887928dea571100d [CANDIDATE]: Term 1 election: Requested vote from peers c971e95cdaec47a18163abd42a952013 (127.0.97.2:42301), 217beb0b90284b199f5b2959b8de96d8 (127.0.97.1:41027)
I20260924 10:00:31.466849 18904 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26b6f8c945c541f09657d90d3edadb6e" candidate_uuid: "5902a70ef84d41f3887928dea571100d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c971e95cdaec47a18163abd42a952013"
I20260924 10:00:31.466885 18770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26b6f8c945c541f09657d90d3edadb6e" candidate_uuid: "5902a70ef84d41f3887928dea571100d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "217beb0b90284b199f5b2959b8de96d8"
I20260924 10:00:31.467010 18904 raft_consensus.cc:3037] T 26b6f8c945c541f09657d90d3edadb6e P c971e95cdaec47a18163abd42a952013 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:00:31.467067 18770 raft_consensus.cc:3037] T 26b6f8c945c541f09657d90d3edadb6e P 217beb0b90284b199f5b2959b8de96d8 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:00:31.469355 18904 raft_consensus.cc:2454] T 26b6f8c945c541f09657d90d3edadb6e P c971e95cdaec47a18163abd42a952013 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5902a70ef84d41f3887928dea571100d in term 1.
I20260924 10:00:31.469347 18770 raft_consensus.cc:2454] T 26b6f8c945c541f09657d90d3edadb6e P 217beb0b90284b199f5b2959b8de96d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5902a70ef84d41f3887928dea571100d in term 1.
I20260924 10:00:31.469683 18973 leader_election.cc:302] T 26b6f8c945c541f09657d90d3edadb6e P 5902a70ef84d41f3887928dea571100d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5902a70ef84d41f3887928dea571100d, c971e95cdaec47a18163abd42a952013; no voters:
I20260924 10:00:31.469951 19108 raft_consensus.cc:2781] T 26b6f8c945c541f09657d90d3edadb6e P 5902a70ef84d41f3887928dea571100d [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:00:31.470415 19108 raft_consensus.cc:686] T 26b6f8c945c541f09657d90d3edadb6e P 5902a70ef84d41f3887928dea571100d [term 1 LEADER]: Becoming Leader. State: Replica: 5902a70ef84d41f3887928dea571100d, State: Running, Role: LEADER
I20260924 10:00:31.470846 19108 consensus_queue.cc:237] T 26b6f8c945c541f09657d90d3edadb6e P 5902a70ef84d41f3887928dea571100d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c971e95cdaec47a18163abd42a952013" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42301 } } peers { permanent_uuid: "217beb0b90284b199f5b2959b8de96d8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41027 } } peers { permanent_uuid: "5902a70ef84d41f3887928dea571100d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45681 } }
I20260924 10:00:31.475200 18625 catalog_manager.cc:5495] T 26b6f8c945c541f09657d90d3edadb6e P 5902a70ef84d41f3887928dea571100d reported cstate change: term changed from 0 to 1, leader changed from <none> to 5902a70ef84d41f3887928dea571100d (127.0.97.3). New cstate: current_term: 1 leader_uuid: "5902a70ef84d41f3887928dea571100d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c971e95cdaec47a18163abd42a952013" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42301 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "217beb0b90284b199f5b2959b8de96d8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5902a70ef84d41f3887928dea571100d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45681 } health_report { overall_health: HEALTHY } } }
W20260924 10:00:31.484928 18950 tablet.cc:2369] T 26b6f8c945c541f09657d90d3edadb6e P c971e95cdaec47a18163abd42a952013: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 10:00:31.514998 18816 tablet.cc:2369] T 26b6f8c945c541f09657d90d3edadb6e P 217beb0b90284b199f5b2959b8de96d8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 10:00:31.527457 19099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:31.522175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39304 (local address 127.0.97.3:45681)
0924 10:00:31.522358 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0924 10:00:31.522364 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:00:31.522456 (+ 92us) server_negotiation.cc:407] Connection header received
0924 10:00:31.522658 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:00:31.522661 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:00:31.522752 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:00:31.522885 (+ 133us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:00:31.523294 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:00:31.524002 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:00:31.525266 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:00:31.525705 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:00:31.525833 (+ 128us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:00:31.526376 (+ 543us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:00:31.526891 (+ 515us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:00:31.527047 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0924 10:00:31.527183 (+ 136us) server_negotiation.cc:299] Negotiation successful
0924 10:00:31.527228 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 10:00:31.534910 18770 raft_consensus.cc:1261] T 26b6f8c945c541f09657d90d3edadb6e P 217beb0b90284b199f5b2959b8de96d8 [term 1 FOLLOWER]: Refusing update from remote peer 5902a70ef84d41f3887928dea571100d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:00:31.534904 18904 raft_consensus.cc:1261] T 26b6f8c945c541f09657d90d3edadb6e P c971e95cdaec47a18163abd42a952013 [term 1 FOLLOWER]: Refusing update from remote peer 5902a70ef84d41f3887928dea571100d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:00:31.535770 19110 consensus_queue.cc:1035] T 26b6f8c945c541f09657d90d3edadb6e P 5902a70ef84d41f3887928dea571100d [LEADER]: Connected to new peer: Peer: permanent_uuid: "217beb0b90284b199f5b2959b8de96d8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:00:31.536135 19108 consensus_queue.cc:1035] T 26b6f8c945c541f09657d90d3edadb6e P 5902a70ef84d41f3887928dea571100d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c971e95cdaec47a18163abd42a952013" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:00:31.550412 19115 mvcc.cc:205] Tried to move back new op lower bound from 7332839553158832128 to 7332839552908783616. Current Snapshot: MvccSnapshot[applied={T|T < 7332839553158832128}]
I20260924 10:00:31.560189 18625 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54296:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:00:31.560433 18625 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54296:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:00:31.563586 18625 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1eefa496903e4b40a841f91d78b12674: Sending DeleteTablet for 3 replicas of tablet 26b6f8c945c541f09657d90d3edadb6e
I20260924 10:00:31.564553 18884 tablet_service.cc:1514] Processing DeleteTablet for tablet 26b6f8c945c541f09657d90d3edadb6e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:00:31 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:56706
I20260924 10:00:31.564663 18750 tablet_service.cc:1514] Processing DeleteTablet for tablet 26b6f8c945c541f09657d90d3edadb6e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:00:31 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:45624
I20260924 10:00:31.565088 19017 tablet_service.cc:1514] Processing DeleteTablet for tablet 26b6f8c945c541f09657d90d3edadb6e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:00:31 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:39294
I20260924 10:00:31.567126 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu pid 18687
I20260924 10:00:31.579914 19122 tablet_replica.cc:330] T 26b6f8c945c541f09657d90d3edadb6e P c971e95cdaec47a18163abd42a952013: stopping tablet replica
I20260924 10:00:31.580448 19122 raft_consensus.cc:2229] T 26b6f8c945c541f09657d90d3edadb6e P c971e95cdaec47a18163abd42a952013 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:00:31.580999 19122 raft_consensus.cc:2258] T 26b6f8c945c541f09657d90d3edadb6e P c971e95cdaec47a18163abd42a952013 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:00:31.583061 19122 ts_tablet_manager.cc:1905] T 26b6f8c945c541f09657d90d3edadb6e P c971e95cdaec47a18163abd42a952013: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:00:31.584930 19123 tablet_replica.cc:330] T 26b6f8c945c541f09657d90d3edadb6e P 217beb0b90284b199f5b2959b8de96d8: stopping tablet replica
I20260924 10:00:31.585633 19124 tablet_replica.cc:330] T 26b6f8c945c541f09657d90d3edadb6e P 5902a70ef84d41f3887928dea571100d: stopping tablet replica
I20260924 10:00:31.586061 19124 raft_consensus.cc:2229] T 26b6f8c945c541f09657d90d3edadb6e P 5902a70ef84d41f3887928dea571100d [term 1 LEADER]: Raft consensus shutting down.
I20260924 10:00:31.586705 19124 raft_consensus.cc:2258] T 26b6f8c945c541f09657d90d3edadb6e P 5902a70ef84d41f3887928dea571100d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:00:31.587363 19123 raft_consensus.cc:2229] T 26b6f8c945c541f09657d90d3edadb6e P 217beb0b90284b199f5b2959b8de96d8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:00:31.587855 19123 raft_consensus.cc:2258] T 26b6f8c945c541f09657d90d3edadb6e P 217beb0b90284b199f5b2959b8de96d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:00:31.589766 19122 ts_tablet_manager.cc:1918] T 26b6f8c945c541f09657d90d3edadb6e P c971e95cdaec47a18163abd42a952013: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:00:31.589916 19122 log.cc:1196] T 26b6f8c945c541f09657d90d3edadb6e P c971e95cdaec47a18163abd42a952013: Deleting WAL directory at /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790243735634835-388-0/minicluster-data/ts-1/wal/wals/26b6f8c945c541f09657d90d3edadb6e
I20260924 10:00:31.590461 19122 ts_tablet_manager.cc:1939] T 26b6f8c945c541f09657d90d3edadb6e P c971e95cdaec47a18163abd42a952013: Deleting consensus metadata
I20260924 10:00:31.591526 19124 ts_tablet_manager.cc:1905] T 26b6f8c945c541f09657d90d3edadb6e P 5902a70ef84d41f3887928dea571100d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:00:31.592242 18613 catalog_manager.cc:4841] TS c971e95cdaec47a18163abd42a952013 (127.0.97.2:42301): tablet 26b6f8c945c541f09657d90d3edadb6e (table test-table [id=1bb20ae5fabc4665be354f212bbe7147]) successfully deleted
Sep 24 10:00:31 dist-test-slave-w8d2 krb5kdc[18581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244029, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260924 10:00:31.595834 19123 ts_tablet_manager.cc:1905] T 26b6f8c945c541f09657d90d3edadb6e P 217beb0b90284b199f5b2959b8de96d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:00:31.598469 19124 ts_tablet_manager.cc:1918] T 26b6f8c945c541f09657d90d3edadb6e P 5902a70ef84d41f3887928dea571100d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:00:31.598629 19124 log.cc:1196] T 26b6f8c945c541f09657d90d3edadb6e P 5902a70ef84d41f3887928dea571100d: Deleting WAL directory at /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790243735634835-388-0/minicluster-data/ts-2/wal/wals/26b6f8c945c541f09657d90d3edadb6e
I20260924 10:00:31.599128 19124 ts_tablet_manager.cc:1939] T 26b6f8c945c541f09657d90d3edadb6e P 5902a70ef84d41f3887928dea571100d: Deleting consensus metadata
I20260924 10:00:31.600795 18613 catalog_manager.cc:4841] TS 5902a70ef84d41f3887928dea571100d (127.0.97.3:45681): tablet 26b6f8c945c541f09657d90d3edadb6e (table test-table [id=1bb20ae5fabc4665be354f212bbe7147]) successfully deleted
I20260924 10:00:31.600898 19123 ts_tablet_manager.cc:1918] T 26b6f8c945c541f09657d90d3edadb6e P 217beb0b90284b199f5b2959b8de96d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:00:31.601034 19123 log.cc:1196] T 26b6f8c945c541f09657d90d3edadb6e P 217beb0b90284b199f5b2959b8de96d8: Deleting WAL directory at /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790243735634835-388-0/minicluster-data/ts-0/wal/wals/26b6f8c945c541f09657d90d3edadb6e
I20260924 10:00:31.601394 19123 ts_tablet_manager.cc:1939] T 26b6f8c945c541f09657d90d3edadb6e P 217beb0b90284b199f5b2959b8de96d8: Deleting consensus metadata
I20260924 10:00:31.602628 18613 catalog_manager.cc:4841] TS 217beb0b90284b199f5b2959b8de96d8 (127.0.97.1:41027): tablet 26b6f8c945c541f09657d90d3edadb6e (table test-table [id=1bb20ae5fabc4665be354f212bbe7147]) successfully deleted
I20260924 10:00:31.604956 19097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:31.569596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45626 (local address 127.0.97.1:41027)
0924 10:00:31.569765 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0924 10:00:31.569772 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:00:31.582440 (+ 12668us) server_negotiation.cc:407] Connection header received
0924 10:00:31.582908 (+ 468us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:00:31.582913 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:00:31.583019 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:00:31.583096 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:00:31.584895 (+ 1799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:00:31.586015 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:00:31.588108 (+ 2093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:00:31.588757 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:00:31.588847 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:00:31.593976 (+ 5129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:00:31.593987 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:00:31.593999 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:00:31.594031 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:00:31.597329 (+ 3298us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:00:31.603068 (+ 5739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:00:31.603074 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:00:31.603076 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:00:31.603202 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:00:31.603548 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:00:31.603552 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:00:31.603554 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:00:31.603875 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:00:31.604126 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0924 10:00:31.604443 (+ 317us) server_negotiation.cc:299] Negotiation successful
0924 10:00:31.604656 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 10:00:31.605473 18810 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:00:31.695830 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu with pid 18687
I20260924 10:00:31.716169 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu pid 18821
Sep 24 10:00:31 dist-test-slave-w8d2 krb5kdc[18581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244029, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260924 10:00:31.732545 19095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:31.716975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56712 (local address 127.0.97.2:42301)
0924 10:00:31.717167 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0924 10:00:31.717173 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:00:31.717194 (+ 21us) server_negotiation.cc:407] Connection header received
0924 10:00:31.717261 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:00:31.717266 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:00:31.717355 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:00:31.717462 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:00:31.718587 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:00:31.719829 (+ 1242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:00:31.720982 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:00:31.721607 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:00:31.721729 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:00:31.725673 (+ 3944us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:00:31.725692 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:00:31.725711 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:00:31.725762 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:00:31.729384 (+ 3622us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:00:31.730134 (+ 750us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:00:31.730144 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:00:31.730148 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:00:31.730301 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:00:31.730640 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:00:31.730645 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:00:31.730648 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:00:31.731155 (+ 507us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:00:31.731471 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0924 10:00:31.731962 (+ 491us) server_negotiation.cc:299] Negotiation successful
0924 10:00:31.732186 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 10:00:31.733320 18944 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:00:31.819958 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu with pid 18821
I20260924 10:00:31.839391 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu pid 18954
Sep 24 10:00:31 dist-test-slave-w8d2 krb5kdc[18581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244029, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260924 10:00:31.856470 19099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:31.840125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39320 (local address 127.0.97.3:45681)
0924 10:00:31.840295 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0924 10:00:31.840302 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:00:31.840510 (+ 208us) server_negotiation.cc:407] Connection header received
0924 10:00:31.840692 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:00:31.840696 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:00:31.840759 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:00:31.840852 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:00:31.842515 (+ 1663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:00:31.843449 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:00:31.845255 (+ 1806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:00:31.845664 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:00:31.845750 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:00:31.851068 (+ 5318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:00:31.851082 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:00:31.851101 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:00:31.851141 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:00:31.853910 (+ 2769us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:00:31.854512 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:00:31.854519 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:00:31.854522 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:00:31.854680 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:00:31.855022 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:00:31.855026 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:00:31.855028 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:00:31.855395 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:00:31.855717 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0924 10:00:31.855912 (+ 195us) server_negotiation.cc:299] Negotiation successful
0924 10:00:31.856115 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 10:00:31.857199 19077 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:00:31.976112 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu with pid 18954
I20260924 10:00:31.991406 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu pid 18594
I20260924 10:00:31.992429 18656 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:00:32.100762 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu with pid 18594
2026-09-24T10:00:32Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4039 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-24T10:00:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:00:32Z Disabled control of system clock
I20260924 10:00:32.166564 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
/tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790243735634835-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790243735634835-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790243735634835-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790243735634835-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37017
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:34163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37017
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 10:00:32.478701 19133 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:00:32.478974 19133 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:00:32.479089 19133 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:00:32.486268 19133 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:00:32.486375 19133 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:00:32.486434 19133 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:00:32.486509 19133 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:00:32.486567 19133 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:00:32.495648 19133 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790243735634835-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790243735634835-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37017
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37017
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790243735634835-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790243735634835-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:00:32.497004 19133 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:00:32.498759 19133 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:00:32.510344 19138 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:00:32.510389 19141 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:00:32.510486 19139 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:00:32.511080 19133 server_base.cc:1034] running on GCE node
I20260924 10:00:32.511910 19133 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:00:32.513341 19133 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:00:32.514554 19133 hybrid_clock.cc:648] HybridClock initialized: now 1790244032514502 us; error 44 us; skew 500 ppm
I20260924 10:00:32.514911 19133 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 10:00:32.517343 19133 webserver.cc:460] Webserver started at http://127.0.97.62:41859/ using document root <none> and password file <none>
I20260924 10:00:32.518011 19133 fs_manager.cc:359] Metadata directory not provided
I20260924 10:00:32.518110 19133 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:00:32.518428 19133 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:00:32.521587 19133 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790243735634835-388-0/minicluster-data/master-0/data/instance:
uuid: "f5498aee23954fb2ab2bfbd6309b0034"
format_stamp: "Formatted at 2026-09-24 10:00:32 on dist-test-slave-w8d2"
server_key: "56d48553c6e7be9c2e57350245e7eb3c"
server_key_iv: "c526e8dbffd4dc2355cdae8aad164bb8"
server_key_version: "encryptionkey@0"
I20260924 10:00:32.522468 19133 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790243735634835-388-0/minicluster-data/master-0/wal/instance:
uuid: "f5498aee23954fb2ab2bfbd6309b0034"
format_stamp: "Formatted at 2026-09-24 10:00:32 on dist-test-slave-w8d2"
server_key: "56d48553c6e7be9c2e57350245e7eb3c"
server_key_iv: "c526e8dbffd4dc2355cdae8aad164bb8"
server_key_version: "encryptionkey@0"
I20260924 10:00:32.527675 19133 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260924 10:00:32.531055 19147 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:00:32.532472 19133 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 10:00:32.532644 19133 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790243735634835-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790243735634835-388-0/minicluster-data/master-0/wal
uuid: "f5498aee23954fb2ab2bfbd6309b0034"
format_stamp: "Formatted at 2026-09-24 10:00:32 on dist-test-slave-w8d2"
server_key: "56d48553c6e7be9c2e57350245e7eb3c"
server_key_iv: "c526e8dbffd4dc2355cdae8aad164bb8"
server_key_version: "encryptionkey@0"
I20260924 10:00:32.532841 19133 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790243735634835-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790243735634835-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790243735634835-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:00:32.565753 19133 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:00:32.566609 19133 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:00:32.566876 19133 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:00:32.584040 19133 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:37017
I20260924 10:00:32.584062 19198 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:37017 every 8 connection(s)
I20260924 10:00:32.585330 19133 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790243735634835-388-0/minicluster-data/master-0/data/info.pb
I20260924 10:00:32.587922 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu as pid 19133
I20260924 10:00:32.588069 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790243735634835-388-0/minicluster-data/master-0/wal/instance
I20260924 10:00:32.588506 388 external_mini_cluster.cc:1428] Setting key 7cfeaf79eccd94b6047d1f286fcdc116
I20260924 10:00:32.588999 19199 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:00:32.600875 19199 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f5498aee23954fb2ab2bfbd6309b0034: Bootstrap starting.
I20260924 10:00:32.601565 19202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:32.590268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41946 (local address 127.0.97.62:37017)
0924 10:00:32.591190 (+ 922us) server_negotiation.cc:206] Beginning negotiation
0924 10:00:32.591201 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 10:00:32.591276 (+ 75us) server_negotiation.cc:407] Connection header received
0924 10:00:32.593521 (+ 2245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:00:32.593542 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:00:32.593995 (+ 453us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:00:32.594370 (+ 375us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:00:32.594920 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:00:32.596214 (+ 1294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:00:32.597498 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:00:32.598160 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:00:32.598346 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:00:32.599423 (+ 1077us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:00:32.599431 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:00:32.599445 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:00:32.599956 (+ 511us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:00:32.600030 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:00:32.600050 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:00:32.600144 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0924 10:00:32.600355 (+ 211us) server_negotiation.cc:299] Negotiation successful
0924 10:00:32.600642 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":527,"thread_start_us":318,"threads_started":1}
I20260924 10:00:32.603956 19199 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f5498aee23954fb2ab2bfbd6309b0034: Neither blocks nor log segments found. Creating new log.
I20260924 10:00:32.604934 19199 log.cc:824] T 00000000000000000000000000000000 P f5498aee23954fb2ab2bfbd6309b0034: Log is configured to *not* fsync() on all Append() calls
I20260924 10:00:32.607401 19199 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f5498aee23954fb2ab2bfbd6309b0034: No bootstrap required, opened a new log
I20260924 10:00:32.614430 19199 raft_consensus.cc:348] T 00000000000000000000000000000000 P f5498aee23954fb2ab2bfbd6309b0034 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5498aee23954fb2ab2bfbd6309b0034" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37017 } }
I20260924 10:00:32.614742 19199 raft_consensus.cc:374] T 00000000000000000000000000000000 P f5498aee23954fb2ab2bfbd6309b0034 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:00:32.614813 19199 raft_consensus.cc:729] T 00000000000000000000000000000000 P f5498aee23954fb2ab2bfbd6309b0034 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5498aee23954fb2ab2bfbd6309b0034, State: Initialized, Role: FOLLOWER
I20260924 10:00:32.615314 19199 consensus_queue.cc:260] T 00000000000000000000000000000000 P f5498aee23954fb2ab2bfbd6309b0034 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5498aee23954fb2ab2bfbd6309b0034" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37017 } }
I20260924 10:00:32.615504 19199 raft_consensus.cc:388] T 00000000000000000000000000000000 P f5498aee23954fb2ab2bfbd6309b0034 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:00:32.615600 19199 raft_consensus.cc:482] T 00000000000000000000000000000000 P f5498aee23954fb2ab2bfbd6309b0034 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:00:32.615737 19199 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f5498aee23954fb2ab2bfbd6309b0034 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:00:32.618005 19199 raft_consensus.cc:504] T 00000000000000000000000000000000 P f5498aee23954fb2ab2bfbd6309b0034 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5498aee23954fb2ab2bfbd6309b0034" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37017 } }
I20260924 10:00:32.618435 19199 leader_election.cc:302] T 00000000000000000000000000000000 P f5498aee23954fb2ab2bfbd6309b0034 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f5498aee23954fb2ab2bfbd6309b0034; no voters:
I20260924 10:00:32.618937 19199 leader_election.cc:288] T 00000000000000000000000000000000 P f5498aee23954fb2ab2bfbd6309b0034 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:00:32.619059 19204 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f5498aee23954fb2ab2bfbd6309b0034 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:00:32.619740 19204 raft_consensus.cc:686] T 00000000000000000000000000000000 P f5498aee23954fb2ab2bfbd6309b0034 [term 1 LEADER]: Becoming Leader. State: Replica: f5498aee23954fb2ab2bfbd6309b0034, State: Running, Role: LEADER
I20260924 10:00:32.620205 19204 consensus_queue.cc:237] T 00000000000000000000000000000000 P f5498aee23954fb2ab2bfbd6309b0034 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5498aee23954fb2ab2bfbd6309b0034" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37017 } }
I20260924 10:00:32.620853 19199 sys_catalog.cc:564] T 00000000000000000000000000000000 P f5498aee23954fb2ab2bfbd6309b0034 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:00:32.622654 19206 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5498aee23954fb2ab2bfbd6309b0034 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f5498aee23954fb2ab2bfbd6309b0034. Latest consensus state: current_term: 1 leader_uuid: "f5498aee23954fb2ab2bfbd6309b0034" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5498aee23954fb2ab2bfbd6309b0034" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37017 } } }
I20260924 10:00:32.623631 19206 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5498aee23954fb2ab2bfbd6309b0034 [sys.catalog]: This master's current role is: LEADER
I20260924 10:00:32.623521 19205 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5498aee23954fb2ab2bfbd6309b0034 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f5498aee23954fb2ab2bfbd6309b0034" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5498aee23954fb2ab2bfbd6309b0034" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37017 } } }
I20260924 10:00:32.624372 19205 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5498aee23954fb2ab2bfbd6309b0034 [sys.catalog]: This master's current role is: LEADER
I20260924 10:00:32.628125 19213 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:00:32.633119 19213 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:00:32.641433 19213 catalog_manager.cc:1347] Generated new cluster ID: 54ad8c9728844c4081300026ac55a8c5
I20260924 10:00:32.641539 19213 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:00:32.654212 19213 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:00:32.655993 19213 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:00:32.663465 19213 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f5498aee23954fb2ab2bfbd6309b0034: Generated new TSK 0
I20260924 10:00:32.664438 19213 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 10:00:32.672191 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
/tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790243735634835-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790243735634835-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790243735634835-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790243735634835-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37017
--builtin_ntp_servers=127.0.97.20:34163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 10:00:32.985111 19223 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:00:32.985388 19223 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:00:32.985486 19223 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:00:32.992630 19223 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:00:32.992739 19223 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:00:32.992852 19223 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260924 10:00:33.002460 19223 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790243735634835-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790243735634835-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790243735634835-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37017
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790243735634835-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:00:33.003746 19223 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:00:33.005461 19223 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:00:33.018639 19228 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:00:33.018723 19231 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:00:33.018744 19229 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:00:33.019506 19223 server_base.cc:1034] running on GCE node
I20260924 10:00:33.020061 19223 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:00:33.020980 19223 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:00:33.022174 19223 hybrid_clock.cc:648] HybridClock initialized: now 1790244033022123 us; error 45 us; skew 500 ppm
I20260924 10:00:33.022481 19223 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 10:00:33.024662 19223 webserver.cc:460] Webserver started at http://127.0.97.1:46833/ using document root <none> and password file <none>
I20260924 10:00:33.025323 19223 fs_manager.cc:359] Metadata directory not provided
I20260924 10:00:33.025424 19223 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:00:33.025717 19223 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:00:33.028643 19223 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790243735634835-388-0/minicluster-data/ts-0/data/instance:
uuid: "e2847751493f4cd89942b850685e864c"
format_stamp: "Formatted at 2026-09-24 10:00:33 on dist-test-slave-w8d2"
server_key: "93f7fc25721086cb6217c21f6c0c43e9"
server_key_iv: "8c068fbc8c1bd3e34da1f62c3b6fb095"
server_key_version: "encryptionkey@0"
I20260924 10:00:33.029405 19223 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790243735634835-388-0/minicluster-data/ts-0/wal/instance:
uuid: "e2847751493f4cd89942b850685e864c"
format_stamp: "Formatted at 2026-09-24 10:00:33 on dist-test-slave-w8d2"
server_key: "93f7fc25721086cb6217c21f6c0c43e9"
server_key_iv: "8c068fbc8c1bd3e34da1f62c3b6fb095"
server_key_version: "encryptionkey@0"
I20260924 10:00:33.034358 19223 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260924 10:00:33.037801 19237 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:00:33.039115 19223 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 10:00:33.039302 19223 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790243735634835-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790243735634835-388-0/minicluster-data/ts-0/wal
uuid: "e2847751493f4cd89942b850685e864c"
format_stamp: "Formatted at 2026-09-24 10:00:33 on dist-test-slave-w8d2"
server_key: "93f7fc25721086cb6217c21f6c0c43e9"
server_key_iv: "8c068fbc8c1bd3e34da1f62c3b6fb095"
server_key_version: "encryptionkey@0"
I20260924 10:00:33.039500 19223 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790243735634835-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790243735634835-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790243735634835-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:00:33.051743 19223 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:00:33.052507 19223 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:00:33.052747 19223 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:00:33.053629 19223 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:00:33.055030 19223 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:00:33.055120 19223 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:00:33.055249 19223 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:00:33.055328 19223 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260924 10:00:33.084122 19223 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:43297
I20260924 10:00:33.084193 19349 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:43297 every 8 connection(s)
I20260924 10:00:33.085381 19223 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790243735634835-388-0/minicluster-data/ts-0/data/info.pb
I20260924 10:00:33.093050 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu as pid 19223
I20260924 10:00:33.093197 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790243735634835-388-0/minicluster-data/ts-0/wal/instance
I20260924 10:00:33.093607 388 external_mini_cluster.cc:1428] Setting key b9ddd60f583aace1483de835462669c3
I20260924 10:00:33.096259 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
/tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790243735634835-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790243735634835-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790243735634835-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790243735634835-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37017
--builtin_ntp_servers=127.0.97.20:34163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 10:00:33.097129 19202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:33.087780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:54855 (local address 127.0.97.62:37017)
0924 10:00:33.087951 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0924 10:00:33.087955 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:00:33.089240 (+ 1285us) server_negotiation.cc:407] Connection header received
0924 10:00:33.090990 (+ 1750us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:00:33.090996 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:00:33.091091 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:00:33.091180 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:00:33.092233 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:00:33.092966 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:00:33.094543 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:00:33.094968 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:00:33.095072 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:00:33.096400 (+ 1328us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:00:33.096405 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:00:33.096410 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:00:33.096429 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:00:33.096468 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:00:33.096484 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:00:33.096560 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 10:00:33.096758 (+ 198us) server_negotiation.cc:299] Negotiation successful
0924 10:00:33.096851 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 10:00:33.098093 19352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:33.088235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37017 (local address 127.0.97.1:54855)
0924 10:00:33.089069 (+ 834us) negotiation.cc:107] Waiting for socket to connect
0924 10:00:33.089131 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0924 10:00:33.090705 (+ 1574us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:00:33.091441 (+ 736us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:00:33.091450 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:00:33.091534 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:00:33.092083 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:00:33.092099 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:00:33.093108 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:00:33.093111 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:00:33.094410 (+ 1299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:00:33.094421 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:00:33.095178 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:00:33.095185 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:00:33.095433 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:00:33.096111 (+ 678us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 10:00:33.096114 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:00:33.096152 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 10:00:33.096228 (+ 76us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:00:33.096570 (+ 342us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:00:33.096582 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:00:33.096590 (+ 8us) client_negotiation.cc:769] Sending connection context
0924 10:00:33.096737 (+ 147us) client_negotiation.cc:240] Negotiation successful
0924 10:00:33.097114 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":266,"threads_started":1}
I20260924 10:00:33.099471 19350 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37017
I20260924 10:00:33.099751 19350 heartbeater.cc:464] Registering TS with master...
I20260924 10:00:33.100543 19350 heartbeater.cc:511] Master 127.0.97.62:37017 requested a full tablet report, sending...
I20260924 10:00:33.102607 19164 ts_manager.cc:194] Registered new tserver with Master: e2847751493f4cd89942b850685e864c (127.0.97.1:43297)
I20260924 10:00:33.104357 19164 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.1:54855
W20260924 10:00:33.405967 19353 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:00:33.406270 19353 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:00:33.406385 19353 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:00:33.413885 19353 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:00:33.413992 19353 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:00:33.414132 19353 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260924 10:00:33.423314 19353 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790243735634835-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790243735634835-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790243735634835-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37017
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790243735634835-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:00:33.424556 19353 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:00:33.426260 19353 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:00:33.438951 19359 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:00:33.438979 19358 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:00:33.438910 19361 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:00:33.439656 19353 server_base.cc:1034] running on GCE node
I20260924 10:00:33.440209 19353 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:00:33.441108 19353 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:00:33.442297 19353 hybrid_clock.cc:648] HybridClock initialized: now 1790244033442242 us; error 54 us; skew 500 ppm
I20260924 10:00:33.442588 19353 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 10:00:33.444782 19353 webserver.cc:460] Webserver started at http://127.0.97.2:41281/ using document root <none> and password file <none>
I20260924 10:00:33.445429 19353 fs_manager.cc:359] Metadata directory not provided
I20260924 10:00:33.445533 19353 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:00:33.445838 19353 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:00:33.448585 19353 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790243735634835-388-0/minicluster-data/ts-1/data/instance:
uuid: "31f1cd346ae44d8891c3fa4ccb471422"
format_stamp: "Formatted at 2026-09-24 10:00:33 on dist-test-slave-w8d2"
server_key: "2051bb4ccd14682f01d7cd799401a631"
server_key_iv: "9bbaa5b5785a76a29d581ad726c32674"
server_key_version: "encryptionkey@0"
I20260924 10:00:33.449321 19353 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790243735634835-388-0/minicluster-data/ts-1/wal/instance:
uuid: "31f1cd346ae44d8891c3fa4ccb471422"
format_stamp: "Formatted at 2026-09-24 10:00:33 on dist-test-slave-w8d2"
server_key: "2051bb4ccd14682f01d7cd799401a631"
server_key_iv: "9bbaa5b5785a76a29d581ad726c32674"
server_key_version: "encryptionkey@0"
I20260924 10:00:33.453998 19353 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260924 10:00:33.457149 19367 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:00:33.458343 19353 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 10:00:33.458482 19353 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790243735634835-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790243735634835-388-0/minicluster-data/ts-1/wal
uuid: "31f1cd346ae44d8891c3fa4ccb471422"
format_stamp: "Formatted at 2026-09-24 10:00:33 on dist-test-slave-w8d2"
server_key: "2051bb4ccd14682f01d7cd799401a631"
server_key_iv: "9bbaa5b5785a76a29d581ad726c32674"
server_key_version: "encryptionkey@0"
I20260924 10:00:33.458669 19353 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790243735634835-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790243735634835-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790243735634835-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:00:33.470259 19353 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:00:33.471009 19353 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:00:33.471279 19353 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:00:33.472146 19353 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:00:33.473475 19353 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:00:33.473572 19353 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:00:33.473670 19353 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:00:33.473737 19353 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:00:33.502065 19353 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:44703
I20260924 10:00:33.502105 19479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:44703 every 8 connection(s)
I20260924 10:00:33.503294 19353 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790243735634835-388-0/minicluster-data/ts-1/data/info.pb
I20260924 10:00:33.509095 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu as pid 19353
I20260924 10:00:33.509260 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790243735634835-388-0/minicluster-data/ts-1/wal/instance
I20260924 10:00:33.509809 388 external_mini_cluster.cc:1428] Setting key 0a7b9166e73e42052bfde753be2b8c1b
I20260924 10:00:33.512634 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
/tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790243735634835-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790243735634835-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790243735634835-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790243735634835-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37017
--builtin_ntp_servers=127.0.97.20:34163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 10:00:33.515707 19202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:33.505706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:48555 (local address 127.0.97.62:37017)
0924 10:00:33.505885 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0924 10:00:33.505889 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:00:33.507184 (+ 1295us) server_negotiation.cc:407] Connection header received
0924 10:00:33.508928 (+ 1744us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:00:33.508932 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:00:33.509008 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:00:33.509090 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:00:33.510381 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:00:33.511161 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:00:33.513209 (+ 2048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:00:33.513602 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:00:33.513698 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:00:33.515062 (+ 1364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:00:33.515067 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:00:33.515070 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:00:33.515087 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:00:33.515117 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:00:33.515129 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:00:33.515192 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 10:00:33.515423 (+ 231us) server_negotiation.cc:299] Negotiation successful
0924 10:00:33.515499 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260924 10:00:33.516907 19482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:33.506177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37017 (local address 127.0.97.2:48555)
0924 10:00:33.507037 (+ 860us) negotiation.cc:107] Waiting for socket to connect
0924 10:00:33.507077 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 10:00:33.508662 (+ 1585us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:00:33.509336 (+ 674us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:00:33.509346 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:00:33.509464 (+ 118us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:00:33.510208 (+ 744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:00:33.510233 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:00:33.511331 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:00:33.511335 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:00:33.513080 (+ 1745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:00:33.513093 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:00:33.513815 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:00:33.513822 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:00:33.514089 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:00:33.514734 (+ 645us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 10:00:33.514737 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:00:33.514790 (+ 53us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 10:00:33.514889 (+ 99us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:00:33.515204 (+ 315us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:00:33.515212 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:00:33.515219 (+ 7us) client_negotiation.cc:769] Sending connection context
0924 10:00:33.515417 (+ 198us) client_negotiation.cc:240] Negotiation successful
0924 10:00:33.515750 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":265,"threads_started":1}
I20260924 10:00:33.518122 19480 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37017
I20260924 10:00:33.518393 19480 heartbeater.cc:464] Registering TS with master...
I20260924 10:00:33.519161 19480 heartbeater.cc:511] Master 127.0.97.62:37017 requested a full tablet report, sending...
I20260924 10:00:33.520519 19164 ts_manager.cc:194] Registered new tserver with Master: 31f1cd346ae44d8891c3fa4ccb471422 (127.0.97.2:44703)
I20260924 10:00:33.521543 19164 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.2:48555
W20260924 10:00:33.824095 19483 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:00:33.824364 19483 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:00:33.824455 19483 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:00:33.831595 19483 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:00:33.831692 19483 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:00:33.831804 19483 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260924 10:00:33.840904 19483 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790243735634835-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790243735634835-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790243735634835-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37017
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790243735634835-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:00:33.842079 19483 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:00:33.843797 19483 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:00:33.855726 19489 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:00:33.855772 19488 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:00:33.855782 19491 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:00:33.856787 19483 server_base.cc:1034] running on GCE node
I20260924 10:00:33.857290 19483 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:00:33.858139 19483 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:00:33.859333 19483 hybrid_clock.cc:648] HybridClock initialized: now 1790244033859276 us; error 39 us; skew 500 ppm
I20260924 10:00:33.859629 19483 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 10:00:33.861730 19483 webserver.cc:460] Webserver started at http://127.0.97.3:42759/ using document root <none> and password file <none>
I20260924 10:00:33.862416 19483 fs_manager.cc:359] Metadata directory not provided
I20260924 10:00:33.862507 19483 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:00:33.862808 19483 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:00:33.865571 19483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790243735634835-388-0/minicluster-data/ts-2/data/instance:
uuid: "8445177236ec44cf854b92ebe4b638c5"
format_stamp: "Formatted at 2026-09-24 10:00:33 on dist-test-slave-w8d2"
server_key: "0f18bea11dade8e42364369b554d2ad7"
server_key_iv: "ee7884946189d346bc99b59710af9269"
server_key_version: "encryptionkey@0"
I20260924 10:00:33.866286 19483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790243735634835-388-0/minicluster-data/ts-2/wal/instance:
uuid: "8445177236ec44cf854b92ebe4b638c5"
format_stamp: "Formatted at 2026-09-24 10:00:33 on dist-test-slave-w8d2"
server_key: "0f18bea11dade8e42364369b554d2ad7"
server_key_iv: "ee7884946189d346bc99b59710af9269"
server_key_version: "encryptionkey@0"
I20260924 10:00:33.870975 19483 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260924 10:00:33.874173 19497 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:00:33.875460 19483 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 10:00:33.875589 19483 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790243735634835-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790243735634835-388-0/minicluster-data/ts-2/wal
uuid: "8445177236ec44cf854b92ebe4b638c5"
format_stamp: "Formatted at 2026-09-24 10:00:33 on dist-test-slave-w8d2"
server_key: "0f18bea11dade8e42364369b554d2ad7"
server_key_iv: "ee7884946189d346bc99b59710af9269"
server_key_version: "encryptionkey@0"
I20260924 10:00:33.875774 19483 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790243735634835-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790243735634835-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790243735634835-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:00:33.901355 19483 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:00:33.902199 19483 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:00:33.902455 19483 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:00:33.903479 19483 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:00:33.904940 19483 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:00:33.905021 19483 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:00:33.905119 19483 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:00:33.905185 19483 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:00:33.933670 19483 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:42781
I20260924 10:00:33.933743 19609 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:42781 every 8 connection(s)
I20260924 10:00:33.934891 19483 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790243735634835-388-0/minicluster-data/ts-2/data/info.pb
I20260924 10:00:33.935914 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu as pid 19483
I20260924 10:00:33.936094 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790243735634835-388-0/minicluster-data/ts-2/wal/instance
I20260924 10:00:33.936652 388 external_mini_cluster.cc:1428] Setting key 2532948b3787c2ce094e1cb17f6700fd
I20260924 10:00:33.946640 19202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:33.937669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:49999 (local address 127.0.97.62:37017)
0924 10:00:33.937840 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0924 10:00:33.937845 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:00:33.939351 (+ 1506us) server_negotiation.cc:407] Connection header received
0924 10:00:33.941113 (+ 1762us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:00:33.941117 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:00:33.941180 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:00:33.941257 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:00:33.942256 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:00:33.942942 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:00:33.944449 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:00:33.944830 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:00:33.944918 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:00:33.946060 (+ 1142us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:00:33.946065 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:00:33.946068 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:00:33.946084 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:00:33.946112 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:00:33.946123 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:00:33.946186 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 10:00:33.946348 (+ 162us) server_negotiation.cc:299] Negotiation successful
0924 10:00:33.946420 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 10:00:33.947706 19612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:33.938312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37017 (local address 127.0.97.3:49999)
0924 10:00:33.939193 (+ 881us) negotiation.cc:107] Waiting for socket to connect
0924 10:00:33.939268 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0924 10:00:33.940887 (+ 1619us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:00:33.941412 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:00:33.941419 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:00:33.941509 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:00:33.942074 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:00:33.942110 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:00:33.943056 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:00:33.943059 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:00:33.944337 (+ 1278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:00:33.944348 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:00:33.944986 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:00:33.944991 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:00:33.945185 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:00:33.945774 (+ 589us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 10:00:33.945777 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:00:33.945815 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 10:00:33.945892 (+ 77us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:00:33.946194 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:00:33.946200 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:00:33.946205 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 10:00:33.946335 (+ 130us) client_negotiation.cc:240] Negotiation successful
0924 10:00:33.946613 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":272,"threads_started":1}
I20260924 10:00:33.948864 19610 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37017
I20260924 10:00:33.949141 19610 heartbeater.cc:464] Registering TS with master...
I20260924 10:00:33.949914 19610 heartbeater.cc:511] Master 127.0.97.62:37017 requested a full tablet report, sending...
I20260924 10:00:33.951272 19164 ts_manager.cc:194] Registered new tserver with Master: 8445177236ec44cf854b92ebe4b638c5 (127.0.97.3:42781)
I20260924 10:00:33.952061 19164 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.3:49999
I20260924 10:00:33.963692 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 10:00:33.972633 19202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:33.967418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58716 (local address 127.0.97.62:37017)
0924 10:00:33.967568 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0924 10:00:33.967572 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:00:33.967886 (+ 314us) server_negotiation.cc:407] Connection header received
0924 10:00:33.968011 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:00:33.968014 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:00:33.968108 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:00:33.968183 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:00:33.968574 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:00:33.969296 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:00:33.970553 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:00:33.970941 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:00:33.971027 (+ 86us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:00:33.972057 (+ 1030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:00:33.972062 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:00:33.972065 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:00:33.972081 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:00:33.972111 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:00:33.972122 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:00:33.972229 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0924 10:00:33.972343 (+ 114us) server_negotiation.cc:299] Negotiation successful
0924 10:00:33.972384 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 10:00:33.976406 19164 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58716:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 10:00:33.983197 19164 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 10:00:34.015252 19623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:33.993886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:44703 (local address 127.0.0.1:46940)
0924 10:00:34.005946 (+ 12060us) negotiation.cc:107] Waiting for socket to connect
0924 10:00:34.005998 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0924 10:00:34.006163 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:00:34.007388 (+ 1225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:00:34.007396 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:00:34.007430 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:00:34.007890 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:00:34.007903 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:00:34.010201 (+ 2298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:00:34.010207 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:00:34.012792 (+ 2585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:00:34.012808 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:00:34.014371 (+ 1563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:00:34.014377 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:00:34.014583 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:00:34.014639 (+ 56us) client_negotiation.cc:769] Sending connection context
0924 10:00:34.014742 (+ 103us) client_negotiation.cc:240] Negotiation successful
0924 10:00:34.014918 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11805,"thread_start_us":12490,"threads_started":1}
I20260924 10:00:34.016358 19624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:33.994313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46940 (local address 127.0.97.2:44703)
0924 10:00:34.006709 (+ 12396us) server_negotiation.cc:206] Beginning negotiation
0924 10:00:34.006717 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 10:00:34.006747 (+ 30us) server_negotiation.cc:407] Connection header received
0924 10:00:34.006848 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:00:34.006856 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:00:34.007117 (+ 261us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:00:34.008121 (+ 1004us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:00:34.008397 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:00:34.010037 (+ 1640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:00:34.013046 (+ 3009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:00:34.014153 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:00:34.015489 (+ 1336us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:00:34.015770 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0924 10:00:34.015841 (+ 71us) server_negotiation.cc:299] Negotiation successful
0924 10:00:34.016011 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12240,"thread_start_us":12125,"threads_started":1}
I20260924 10:00:34.016436 19622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:33.994020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43297 (local address 127.0.0.1:38746)
0924 10:00:34.006619 (+ 12599us) negotiation.cc:107] Waiting for socket to connect
0924 10:00:34.006648 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0924 10:00:34.006845 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:00:34.007817 (+ 972us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:00:34.007822 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:00:34.007853 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:00:34.008251 (+ 398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:00:34.008262 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:00:34.011425 (+ 3163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:00:34.011430 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:00:34.013808 (+ 2378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:00:34.013820 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:00:34.015719 (+ 1899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:00:34.015726 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:00:34.015900 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:00:34.015931 (+ 31us) client_negotiation.cc:769] Sending connection context
0924 10:00:34.016015 (+ 84us) client_negotiation.cc:240] Negotiation successful
0924 10:00:34.016140 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12492,"thread_start_us":11541,"threads_started":1}
I20260924 10:00:34.017412 19625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:33.994327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38746 (local address 127.0.97.1:43297)
0924 10:00:34.005197 (+ 10870us) server_negotiation.cc:206] Beginning negotiation
0924 10:00:34.005209 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 10:00:34.006743 (+ 1534us) server_negotiation.cc:407] Connection header received
0924 10:00:34.007055 (+ 312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:00:34.007064 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:00:34.007324 (+ 260us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:00:34.007497 (+ 173us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:00:34.008429 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:00:34.010050 (+ 1621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:00:34.013983 (+ 3933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:00:34.015192 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:00:34.015333 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:00:34.015569 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0924 10:00:34.016893 (+ 1324us) server_negotiation.cc:299] Negotiation successful
0924 10:00:34.017073 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10655,"thread_start_us":10889,"threads_started":1}
I20260924 10:00:34.021139 19285 tablet_service.cc:1467] Processing CreateTablet for tablet a1f783f32eba48aa8a89d48b5e749a77 (DEFAULT_TABLE table=test-table [id=d4c259d12eb940c3aa1eff41e7d143ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:00:34.021629 19415 tablet_service.cc:1467] Processing CreateTablet for tablet a1f783f32eba48aa8a89d48b5e749a77 (DEFAULT_TABLE table=test-table [id=d4c259d12eb940c3aa1eff41e7d143ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:00:34.022363 19285 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1f783f32eba48aa8a89d48b5e749a77. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:00:34.022876 19627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:34.007098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56964 (local address 127.0.97.3:42781)
0924 10:00:34.013666 (+ 6568us) server_negotiation.cc:206] Beginning negotiation
0924 10:00:34.013674 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 10:00:34.014297 (+ 623us) server_negotiation.cc:407] Connection header received
0924 10:00:34.014713 (+ 416us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:00:34.014722 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:00:34.014970 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:00:34.015159 (+ 189us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:00:34.016024 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:00:34.017740 (+ 1716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:00:34.020060 (+ 2320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:00:34.021490 (+ 1430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:00:34.021627 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:00:34.021868 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0924 10:00:34.022363 (+ 495us) server_negotiation.cc:299] Negotiation successful
0924 10:00:34.022513 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6380,"thread_start_us":6230,"threads_started":1}
I20260924 10:00:34.022958 19415 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1f783f32eba48aa8a89d48b5e749a77. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:00:34.023530 19626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:34.006752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:42781 (local address 127.0.0.1:56964)
0924 10:00:34.014192 (+ 7440us) negotiation.cc:107] Waiting for socket to connect
0924 10:00:34.014209 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 10:00:34.014535 (+ 326us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:00:34.015461 (+ 926us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:00:34.015465 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:00:34.015489 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:00:34.015820 (+ 331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:00:34.015832 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:00:34.017994 (+ 2162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:00:34.018000 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:00:34.019872 (+ 1872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:00:34.019886 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:00:34.021964 (+ 2078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:00:34.021970 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:00:34.022132 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:00:34.022163 (+ 31us) client_negotiation.cc:769] Sending connection context
0924 10:00:34.022240 (+ 77us) client_negotiation.cc:240] Negotiation successful
0924 10:00:34.023172 (+ 932us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7314,"mutex_wait_us":1,"thread_start_us":7568,"threads_started":1}
I20260924 10:00:34.026813 19545 tablet_service.cc:1467] Processing CreateTablet for tablet a1f783f32eba48aa8a89d48b5e749a77 (DEFAULT_TABLE table=test-table [id=d4c259d12eb940c3aa1eff41e7d143ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:00:34.028194 19545 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1f783f32eba48aa8a89d48b5e749a77. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:00:34.035100 19628 tablet_bootstrap.cc:492] T a1f783f32eba48aa8a89d48b5e749a77 P 31f1cd346ae44d8891c3fa4ccb471422: Bootstrap starting.
I20260924 10:00:34.037081 19629 tablet_bootstrap.cc:492] T a1f783f32eba48aa8a89d48b5e749a77 P e2847751493f4cd89942b850685e864c: Bootstrap starting.
I20260924 10:00:34.038806 19628 tablet_bootstrap.cc:654] T a1f783f32eba48aa8a89d48b5e749a77 P 31f1cd346ae44d8891c3fa4ccb471422: Neither blocks nor log segments found. Creating new log.
I20260924 10:00:34.040022 19628 log.cc:824] T a1f783f32eba48aa8a89d48b5e749a77 P 31f1cd346ae44d8891c3fa4ccb471422: Log is configured to *not* fsync() on all Append() calls
I20260924 10:00:34.040405 19629 tablet_bootstrap.cc:654] T a1f783f32eba48aa8a89d48b5e749a77 P e2847751493f4cd89942b850685e864c: Neither blocks nor log segments found. Creating new log.
I20260924 10:00:34.040812 19630 tablet_bootstrap.cc:492] T a1f783f32eba48aa8a89d48b5e749a77 P 8445177236ec44cf854b92ebe4b638c5: Bootstrap starting.
I20260924 10:00:34.041616 19629 log.cc:824] T a1f783f32eba48aa8a89d48b5e749a77 P e2847751493f4cd89942b850685e864c: Log is configured to *not* fsync() on all Append() calls
I20260924 10:00:34.044060 19628 tablet_bootstrap.cc:492] T a1f783f32eba48aa8a89d48b5e749a77 P 31f1cd346ae44d8891c3fa4ccb471422: No bootstrap required, opened a new log
I20260924 10:00:34.044054 19630 tablet_bootstrap.cc:654] T a1f783f32eba48aa8a89d48b5e749a77 P 8445177236ec44cf854b92ebe4b638c5: Neither blocks nor log segments found. Creating new log.
I20260924 10:00:34.044245 19629 tablet_bootstrap.cc:492] T a1f783f32eba48aa8a89d48b5e749a77 P e2847751493f4cd89942b850685e864c: No bootstrap required, opened a new log
I20260924 10:00:34.044308 19628 ts_tablet_manager.cc:1397] T a1f783f32eba48aa8a89d48b5e749a77 P 31f1cd346ae44d8891c3fa4ccb471422: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.001s
I20260924 10:00:34.044487 19629 ts_tablet_manager.cc:1397] T a1f783f32eba48aa8a89d48b5e749a77 P e2847751493f4cd89942b850685e864c: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260924 10:00:34.045112 19630 log.cc:824] T a1f783f32eba48aa8a89d48b5e749a77 P 8445177236ec44cf854b92ebe4b638c5: Log is configured to *not* fsync() on all Append() calls
I20260924 10:00:34.047959 19630 tablet_bootstrap.cc:492] T a1f783f32eba48aa8a89d48b5e749a77 P 8445177236ec44cf854b92ebe4b638c5: No bootstrap required, opened a new log
I20260924 10:00:34.048198 19630 ts_tablet_manager.cc:1397] T a1f783f32eba48aa8a89d48b5e749a77 P 8445177236ec44cf854b92ebe4b638c5: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260924 10:00:34.052083 19629 raft_consensus.cc:348] T a1f783f32eba48aa8a89d48b5e749a77 P e2847751493f4cd89942b850685e864c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f1cd346ae44d8891c3fa4ccb471422" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44703 } } peers { permanent_uuid: "e2847751493f4cd89942b850685e864c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43297 } } peers { permanent_uuid: "8445177236ec44cf854b92ebe4b638c5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42781 } }
I20260924 10:00:34.052433 19629 raft_consensus.cc:374] T a1f783f32eba48aa8a89d48b5e749a77 P e2847751493f4cd89942b850685e864c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:00:34.052524 19629 raft_consensus.cc:729] T a1f783f32eba48aa8a89d48b5e749a77 P e2847751493f4cd89942b850685e864c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2847751493f4cd89942b850685e864c, State: Initialized, Role: FOLLOWER
I20260924 10:00:34.053071 19629 consensus_queue.cc:260] T a1f783f32eba48aa8a89d48b5e749a77 P e2847751493f4cd89942b850685e864c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f1cd346ae44d8891c3fa4ccb471422" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44703 } } peers { permanent_uuid: "e2847751493f4cd89942b850685e864c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43297 } } peers { permanent_uuid: "8445177236ec44cf854b92ebe4b638c5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42781 } }
I20260924 10:00:34.054129 19350 heartbeater.cc:503] Master 127.0.97.62:37017 was elected leader, sending a full tablet report...
I20260924 10:00:34.053889 19628 raft_consensus.cc:348] T a1f783f32eba48aa8a89d48b5e749a77 P 31f1cd346ae44d8891c3fa4ccb471422 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f1cd346ae44d8891c3fa4ccb471422" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44703 } } peers { permanent_uuid: "e2847751493f4cd89942b850685e864c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43297 } } peers { permanent_uuid: "8445177236ec44cf854b92ebe4b638c5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42781 } }
I20260924 10:00:34.054275 19628 raft_consensus.cc:374] T a1f783f32eba48aa8a89d48b5e749a77 P 31f1cd346ae44d8891c3fa4ccb471422 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:00:34.054354 19628 raft_consensus.cc:729] T a1f783f32eba48aa8a89d48b5e749a77 P 31f1cd346ae44d8891c3fa4ccb471422 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31f1cd346ae44d8891c3fa4ccb471422, State: Initialized, Role: FOLLOWER
I20260924 10:00:34.054785 19629 ts_tablet_manager.cc:1428] T a1f783f32eba48aa8a89d48b5e749a77 P e2847751493f4cd89942b850685e864c: Time spent starting tablet: real 0.010s user 0.010s sys 0.000s
I20260924 10:00:34.054912 19628 consensus_queue.cc:260] T a1f783f32eba48aa8a89d48b5e749a77 P 31f1cd346ae44d8891c3fa4ccb471422 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f1cd346ae44d8891c3fa4ccb471422" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44703 } } peers { permanent_uuid: "e2847751493f4cd89942b850685e864c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43297 } } peers { permanent_uuid: "8445177236ec44cf854b92ebe4b638c5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42781 } }
I20260924 10:00:34.056510 19628 ts_tablet_manager.cc:1428] T a1f783f32eba48aa8a89d48b5e749a77 P 31f1cd346ae44d8891c3fa4ccb471422: Time spent starting tablet: real 0.012s user 0.010s sys 0.002s
I20260924 10:00:34.056965 19480 heartbeater.cc:503] Master 127.0.97.62:37017 was elected leader, sending a full tablet report...
I20260924 10:00:34.057837 19630 raft_consensus.cc:348] T a1f783f32eba48aa8a89d48b5e749a77 P 8445177236ec44cf854b92ebe4b638c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f1cd346ae44d8891c3fa4ccb471422" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44703 } } peers { permanent_uuid: "e2847751493f4cd89942b850685e864c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43297 } } peers { permanent_uuid: "8445177236ec44cf854b92ebe4b638c5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42781 } }
I20260924 10:00:34.058254 19630 raft_consensus.cc:374] T a1f783f32eba48aa8a89d48b5e749a77 P 8445177236ec44cf854b92ebe4b638c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:00:34.058352 19630 raft_consensus.cc:729] T a1f783f32eba48aa8a89d48b5e749a77 P 8445177236ec44cf854b92ebe4b638c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8445177236ec44cf854b92ebe4b638c5, State: Initialized, Role: FOLLOWER
I20260924 10:00:34.058998 19630 consensus_queue.cc:260] T a1f783f32eba48aa8a89d48b5e749a77 P 8445177236ec44cf854b92ebe4b638c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f1cd346ae44d8891c3fa4ccb471422" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44703 } } peers { permanent_uuid: "e2847751493f4cd89942b850685e864c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43297 } } peers { permanent_uuid: "8445177236ec44cf854b92ebe4b638c5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42781 } }
I20260924 10:00:34.060067 19610 heartbeater.cc:503] Master 127.0.97.62:37017 was elected leader, sending a full tablet report...
I20260924 10:00:34.060480 19630 ts_tablet_manager.cc:1428] T a1f783f32eba48aa8a89d48b5e749a77 P 8445177236ec44cf854b92ebe4b638c5: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
W20260924 10:00:34.087369 19351 tablet.cc:2369] T a1f783f32eba48aa8a89d48b5e749a77 P e2847751493f4cd89942b850685e864c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 10:00:34.186565 19611 tablet.cc:2369] T a1f783f32eba48aa8a89d48b5e749a77 P 8445177236ec44cf854b92ebe4b638c5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 10:00:34.255195 19481 tablet.cc:2369] T a1f783f32eba48aa8a89d48b5e749a77 P 31f1cd346ae44d8891c3fa4ccb471422: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 10:00:34.382004 19635 raft_consensus.cc:482] T a1f783f32eba48aa8a89d48b5e749a77 P 31f1cd346ae44d8891c3fa4ccb471422 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 10:00:34.382340 19635 raft_consensus.cc:504] T a1f783f32eba48aa8a89d48b5e749a77 P 31f1cd346ae44d8891c3fa4ccb471422 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f1cd346ae44d8891c3fa4ccb471422" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44703 } } peers { permanent_uuid: "e2847751493f4cd89942b850685e864c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43297 } } peers { permanent_uuid: "8445177236ec44cf854b92ebe4b638c5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42781 } }
I20260924 10:00:34.384042 19635 leader_election.cc:288] T a1f783f32eba48aa8a89d48b5e749a77 P 31f1cd346ae44d8891c3fa4ccb471422 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e2847751493f4cd89942b850685e864c (127.0.97.1:43297), 8445177236ec44cf854b92ebe4b638c5 (127.0.97.3:42781)
I20260924 10:00:34.390259 19625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:34.384393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:49147 (local address 127.0.97.1:43297)
0924 10:00:34.384588 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0924 10:00:34.384594 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:00:34.385315 (+ 721us) server_negotiation.cc:407] Connection header received
0924 10:00:34.385468 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:00:34.385472 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:00:34.385541 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:00:34.385612 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:00:34.386242 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:00:34.386899 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:00:34.388695 (+ 1796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:00:34.389417 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:00:34.389523 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:00:34.389606 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0924 10:00:34.389910 (+ 304us) server_negotiation.cc:299] Negotiation successful
0924 10:00:34.390031 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260924 10:00:34.390264 19627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:34.384453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:43733 (local address 127.0.97.3:42781)
0924 10:00:34.384630 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0924 10:00:34.384636 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:00:34.385259 (+ 623us) server_negotiation.cc:407] Connection header received
0924 10:00:34.385444 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:00:34.385448 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:00:34.385508 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:00:34.385580 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:00:34.386321 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:00:34.387000 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:00:34.388734 (+ 1734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:00:34.389424 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:00:34.389543 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:00:34.389634 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0924 10:00:34.389923 (+ 289us) server_negotiation.cc:299] Negotiation successful
0924 10:00:34.390040 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 10:00:34.390498 19638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:34.384391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43297 (local address 127.0.97.2:49147)
0924 10:00:34.385209 (+ 818us) negotiation.cc:107] Waiting for socket to connect
0924 10:00:34.385223 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 10:00:34.385340 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:00:34.385636 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:00:34.385640 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:00:34.385660 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:00:34.386093 (+ 433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:00:34.386103 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:00:34.387026 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:00:34.387030 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:00:34.388524 (+ 1494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:00:34.388535 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:00:34.389596 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:00:34.389602 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:00:34.389747 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:00:34.389795 (+ 48us) client_negotiation.cc:769] Sending connection context
0924 10:00:34.389867 (+ 72us) client_negotiation.cc:240] Negotiation successful
0924 10:00:34.389988 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":726,"mutex_wait_us":106,"thread_start_us":460,"threads_started":1}
I20260924 10:00:34.390547 19637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:34.384326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:42781 (local address 127.0.97.2:43733)
0924 10:00:34.385155 (+ 829us) negotiation.cc:107] Waiting for socket to connect
0924 10:00:34.385174 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 10:00:34.385310 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:00:34.385616 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:00:34.385620 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:00:34.385642 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:00:34.386184 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:00:34.386195 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:00:34.387127 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:00:34.387135 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:00:34.388561 (+ 1426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:00:34.388575 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:00:34.389613 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:00:34.389619 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:00:34.389792 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:00:34.389813 (+ 21us) client_negotiation.cc:769] Sending connection context
0924 10:00:34.389885 (+ 72us) client_negotiation.cc:240] Negotiation successful
0924 10:00:34.390013 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":719,"mutex_wait_us":31,"thread_start_us":498,"threads_started":1}
I20260924 10:00:34.391461 19565 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1f783f32eba48aa8a89d48b5e749a77" candidate_uuid: "31f1cd346ae44d8891c3fa4ccb471422" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8445177236ec44cf854b92ebe4b638c5" is_pre_election: true
I20260924 10:00:34.391474 19305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1f783f32eba48aa8a89d48b5e749a77" candidate_uuid: "31f1cd346ae44d8891c3fa4ccb471422" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2847751493f4cd89942b850685e864c" is_pre_election: true
I20260924 10:00:34.391881 19565 raft_consensus.cc:2454] T a1f783f32eba48aa8a89d48b5e749a77 P 8445177236ec44cf854b92ebe4b638c5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31f1cd346ae44d8891c3fa4ccb471422 in term 0.
I20260924 10:00:34.391884 19305 raft_consensus.cc:2454] T a1f783f32eba48aa8a89d48b5e749a77 P e2847751493f4cd89942b850685e864c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31f1cd346ae44d8891c3fa4ccb471422 in term 0.
I20260924 10:00:34.392396 19369 leader_election.cc:302] T a1f783f32eba48aa8a89d48b5e749a77 P 31f1cd346ae44d8891c3fa4ccb471422 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31f1cd346ae44d8891c3fa4ccb471422, e2847751493f4cd89942b850685e864c; no voters:
I20260924 10:00:34.392745 19635 raft_consensus.cc:2781] T a1f783f32eba48aa8a89d48b5e749a77 P 31f1cd346ae44d8891c3fa4ccb471422 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 10:00:34.392865 19635 raft_consensus.cc:482] T a1f783f32eba48aa8a89d48b5e749a77 P 31f1cd346ae44d8891c3fa4ccb471422 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 10:00:34.392961 19635 raft_consensus.cc:3037] T a1f783f32eba48aa8a89d48b5e749a77 P 31f1cd346ae44d8891c3fa4ccb471422 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:00:34.395042 19635 raft_consensus.cc:504] T a1f783f32eba48aa8a89d48b5e749a77 P 31f1cd346ae44d8891c3fa4ccb471422 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f1cd346ae44d8891c3fa4ccb471422" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44703 } } peers { permanent_uuid: "e2847751493f4cd89942b850685e864c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43297 } } peers { permanent_uuid: "8445177236ec44cf854b92ebe4b638c5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42781 } }
I20260924 10:00:34.395613 19635 leader_election.cc:288] T a1f783f32eba48aa8a89d48b5e749a77 P 31f1cd346ae44d8891c3fa4ccb471422 [CANDIDATE]: Term 1 election: Requested vote from peers e2847751493f4cd89942b850685e864c (127.0.97.1:43297), 8445177236ec44cf854b92ebe4b638c5 (127.0.97.3:42781)
I20260924 10:00:34.395963 19305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1f783f32eba48aa8a89d48b5e749a77" candidate_uuid: "31f1cd346ae44d8891c3fa4ccb471422" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2847751493f4cd89942b850685e864c"
I20260924 10:00:34.395985 19565 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1f783f32eba48aa8a89d48b5e749a77" candidate_uuid: "31f1cd346ae44d8891c3fa4ccb471422" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8445177236ec44cf854b92ebe4b638c5"
I20260924 10:00:34.396128 19565 raft_consensus.cc:3037] T a1f783f32eba48aa8a89d48b5e749a77 P 8445177236ec44cf854b92ebe4b638c5 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:00:34.396124 19305 raft_consensus.cc:3037] T a1f783f32eba48aa8a89d48b5e749a77 P e2847751493f4cd89942b850685e864c [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:00:34.398141 19305 raft_consensus.cc:2454] T a1f783f32eba48aa8a89d48b5e749a77 P e2847751493f4cd89942b850685e864c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31f1cd346ae44d8891c3fa4ccb471422 in term 1.
I20260924 10:00:34.398168 19565 raft_consensus.cc:2454] T a1f783f32eba48aa8a89d48b5e749a77 P 8445177236ec44cf854b92ebe4b638c5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31f1cd346ae44d8891c3fa4ccb471422 in term 1.
I20260924 10:00:34.398458 19369 leader_election.cc:302] T a1f783f32eba48aa8a89d48b5e749a77 P 31f1cd346ae44d8891c3fa4ccb471422 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31f1cd346ae44d8891c3fa4ccb471422, e2847751493f4cd89942b850685e864c; no voters:
I20260924 10:00:34.398696 19635 raft_consensus.cc:2781] T a1f783f32eba48aa8a89d48b5e749a77 P 31f1cd346ae44d8891c3fa4ccb471422 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:00:34.399072 19635 raft_consensus.cc:686] T a1f783f32eba48aa8a89d48b5e749a77 P 31f1cd346ae44d8891c3fa4ccb471422 [term 1 LEADER]: Becoming Leader. State: Replica: 31f1cd346ae44d8891c3fa4ccb471422, State: Running, Role: LEADER
I20260924 10:00:34.399540 19635 consensus_queue.cc:237] T a1f783f32eba48aa8a89d48b5e749a77 P 31f1cd346ae44d8891c3fa4ccb471422 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f1cd346ae44d8891c3fa4ccb471422" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44703 } } peers { permanent_uuid: "e2847751493f4cd89942b850685e864c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43297 } } peers { permanent_uuid: "8445177236ec44cf854b92ebe4b638c5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42781 } }
I20260924 10:00:34.403954 19163 catalog_manager.cc:5495] T a1f783f32eba48aa8a89d48b5e749a77 P 31f1cd346ae44d8891c3fa4ccb471422 reported cstate change: term changed from 0 to 1, leader changed from <none> to 31f1cd346ae44d8891c3fa4ccb471422 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "31f1cd346ae44d8891c3fa4ccb471422" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f1cd346ae44d8891c3fa4ccb471422" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44703 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e2847751493f4cd89942b850685e864c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43297 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8445177236ec44cf854b92ebe4b638c5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42781 } health_report { overall_health: UNKNOWN } } }
I20260924 10:00:34.458499 19624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:34.453999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46952 (local address 127.0.97.2:44703)
0924 10:00:34.454146 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0924 10:00:34.454150 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:00:34.454167 (+ 17us) server_negotiation.cc:407] Connection header received
0924 10:00:34.454246 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:00:34.454250 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:00:34.454323 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:00:34.454446 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:00:34.454746 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:00:34.455496 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:00:34.456709 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:00:34.457102 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:00:34.457187 (+ 85us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:00:34.457587 (+ 400us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:00:34.458063 (+ 476us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:00:34.458196 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0924 10:00:34.458248 (+ 52us) server_negotiation.cc:299] Negotiation successful
0924 10:00:34.458285 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 10:00:34.466715 19305 raft_consensus.cc:1261] T a1f783f32eba48aa8a89d48b5e749a77 P e2847751493f4cd89942b850685e864c [term 1 FOLLOWER]: Refusing update from remote peer 31f1cd346ae44d8891c3fa4ccb471422: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:00:34.466715 19565 raft_consensus.cc:1261] T a1f783f32eba48aa8a89d48b5e749a77 P 8445177236ec44cf854b92ebe4b638c5 [term 1 FOLLOWER]: Refusing update from remote peer 31f1cd346ae44d8891c3fa4ccb471422: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:00:34.467607 19639 consensus_queue.cc:1035] T a1f783f32eba48aa8a89d48b5e749a77 P 31f1cd346ae44d8891c3fa4ccb471422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2847751493f4cd89942b850685e864c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:00:34.467919 19635 consensus_queue.cc:1035] T a1f783f32eba48aa8a89d48b5e749a77 P 31f1cd346ae44d8891c3fa4ccb471422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8445177236ec44cf854b92ebe4b638c5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:00:34.479024 19642 mvcc.cc:205] Tried to move back new op lower bound from 7332839565165768704 to 7332839564904636416. Current Snapshot: MvccSnapshot[applied={T|T < 7332839565165768704}]
I20260924 10:00:34.494414 19164 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58716:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:00:34.494647 19164 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58716:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:00:34.497578 19164 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f5498aee23954fb2ab2bfbd6309b0034: Sending DeleteTablet for 3 replicas of tablet a1f783f32eba48aa8a89d48b5e749a77
I20260924 10:00:34.498567 19415 tablet_service.cc:1514] Processing DeleteTablet for tablet a1f783f32eba48aa8a89d48b5e749a77 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:00:34 UTC) from {username='slave'} at 127.0.0.1:46940
I20260924 10:00:34.498627 19285 tablet_service.cc:1514] Processing DeleteTablet for tablet a1f783f32eba48aa8a89d48b5e749a77 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:00:34 UTC) from {username='slave'} at 127.0.0.1:38746
I20260924 10:00:34.499001 19545 tablet_service.cc:1514] Processing DeleteTablet for tablet a1f783f32eba48aa8a89d48b5e749a77 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:00:34 UTC) from {username='slave'} at 127.0.0.1:56964
I20260924 10:00:34.499382 19653 tablet_replica.cc:330] T a1f783f32eba48aa8a89d48b5e749a77 P e2847751493f4cd89942b850685e864c: stopping tablet replica
I20260924 10:00:34.499756 19653 raft_consensus.cc:2229] T a1f783f32eba48aa8a89d48b5e749a77 P e2847751493f4cd89942b850685e864c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:00:34.500105 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu pid 19223
I20260924 10:00:34.500205 19653 raft_consensus.cc:2258] T a1f783f32eba48aa8a89d48b5e749a77 P e2847751493f4cd89942b850685e864c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:00:34.500955 19652 tablet_replica.cc:330] T a1f783f32eba48aa8a89d48b5e749a77 P 31f1cd346ae44d8891c3fa4ccb471422: stopping tablet replica
I20260924 10:00:34.501327 19652 raft_consensus.cc:2229] T a1f783f32eba48aa8a89d48b5e749a77 P 31f1cd346ae44d8891c3fa4ccb471422 [term 1 LEADER]: Raft consensus shutting down.
I20260924 10:00:34.502007 19652 raft_consensus.cc:2258] T a1f783f32eba48aa8a89d48b5e749a77 P 31f1cd346ae44d8891c3fa4ccb471422 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:00:34.504055 19652 ts_tablet_manager.cc:1905] T a1f783f32eba48aa8a89d48b5e749a77 P 31f1cd346ae44d8891c3fa4ccb471422: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:00:34.506532 19653 ts_tablet_manager.cc:1905] T a1f783f32eba48aa8a89d48b5e749a77 P e2847751493f4cd89942b850685e864c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:00:34.511619 19652 ts_tablet_manager.cc:1918] T a1f783f32eba48aa8a89d48b5e749a77 P 31f1cd346ae44d8891c3fa4ccb471422: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:00:34.511767 19652 log.cc:1196] T a1f783f32eba48aa8a89d48b5e749a77 P 31f1cd346ae44d8891c3fa4ccb471422: Deleting WAL directory at /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790243735634835-388-0/minicluster-data/ts-1/wal/wals/a1f783f32eba48aa8a89d48b5e749a77
I20260924 10:00:34.512260 19652 ts_tablet_manager.cc:1939] T a1f783f32eba48aa8a89d48b5e749a77 P 31f1cd346ae44d8891c3fa4ccb471422: Deleting consensus metadata
I20260924 10:00:34.513844 19150 catalog_manager.cc:4841] TS 31f1cd346ae44d8891c3fa4ccb471422 (127.0.97.2:44703): tablet a1f783f32eba48aa8a89d48b5e749a77 (table test-table [id=d4c259d12eb940c3aa1eff41e7d143ab]) successfully deleted
I20260924 10:00:34.518666 19625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:34.502346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38750 (local address 127.0.97.1:43297)
0924 10:00:34.502516 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0924 10:00:34.502522 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:00:34.512270 (+ 9748us) server_negotiation.cc:407] Connection header received
0924 10:00:34.512799 (+ 529us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:00:34.512804 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:00:34.512903 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:00:34.512977 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:00:34.513641 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:00:34.514527 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:00:34.515891 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:00:34.516311 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:00:34.516413 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:00:34.517448 (+ 1035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:00:34.517456 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:00:34.517468 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:00:34.517968 (+ 500us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:00:34.518018 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:00:34.518036 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:00:34.518126 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0924 10:00:34.518238 (+ 112us) server_negotiation.cc:299] Negotiation successful
0924 10:00:34.518338 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 10:00:34.518666 19654 tablet_replica.cc:330] T a1f783f32eba48aa8a89d48b5e749a77 P 8445177236ec44cf854b92ebe4b638c5: stopping tablet replica
I20260924 10:00:34.519037 19654 raft_consensus.cc:2229] T a1f783f32eba48aa8a89d48b5e749a77 P 8445177236ec44cf854b92ebe4b638c5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:00:34.519316 19345 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:00:34.519518 19654 raft_consensus.cc:2258] T a1f783f32eba48aa8a89d48b5e749a77 P 8445177236ec44cf854b92ebe4b638c5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:00:34.524602 19654 ts_tablet_manager.cc:1905] T a1f783f32eba48aa8a89d48b5e749a77 P 8445177236ec44cf854b92ebe4b638c5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:00:34.531428 19654 ts_tablet_manager.cc:1918] T a1f783f32eba48aa8a89d48b5e749a77 P 8445177236ec44cf854b92ebe4b638c5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:00:34.531581 19654 log.cc:1196] T a1f783f32eba48aa8a89d48b5e749a77 P 8445177236ec44cf854b92ebe4b638c5: Deleting WAL directory at /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790243735634835-388-0/minicluster-data/ts-2/wal/wals/a1f783f32eba48aa8a89d48b5e749a77
I20260924 10:00:34.532044 19654 ts_tablet_manager.cc:1939] T a1f783f32eba48aa8a89d48b5e749a77 P 8445177236ec44cf854b92ebe4b638c5: Deleting consensus metadata
I20260924 10:00:34.533756 19150 catalog_manager.cc:4841] TS 8445177236ec44cf854b92ebe4b638c5 (127.0.97.3:42781): tablet a1f783f32eba48aa8a89d48b5e749a77 (table test-table [id=d4c259d12eb940c3aa1eff41e7d143ab]) successfully deleted
I20260924 10:00:34.641741 19653 ts_tablet_manager.cc:1918] T a1f783f32eba48aa8a89d48b5e749a77 P e2847751493f4cd89942b850685e864c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:00:34.641989 19653 log.cc:1196] T a1f783f32eba48aa8a89d48b5e749a77 P e2847751493f4cd89942b850685e864c: Deleting WAL directory at /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790243735634835-388-0/minicluster-data/ts-0/wal/wals/a1f783f32eba48aa8a89d48b5e749a77
I20260924 10:00:34.642408 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu with pid 19223
W20260924 10:00:34.663005 19149 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:43297 (error 108)
I20260924 10:00:34.663148 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu pid 19353
I20260924 10:00:34.666069 19658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:34.664883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43297 (local address 127.0.0.1:38754)
0924 10:00:34.665813 (+ 930us) negotiation.cc:107] Waiting for socket to connect
0924 10:00:34.665916 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:43297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":799,"thread_start_us":725,"threads_started":1}
W20260924 10:00:34.666287 19149 catalog_manager.cc:4584] TS e2847751493f4cd89942b850685e864c (127.0.97.1:43297): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a1f783f32eba48aa8a89d48b5e749a77: Network error: Client connection negotiation failed: client connection to 127.0.97.1:43297: connect: Connection refused (error 111)
I20260924 10:00:34.671406 19624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:34.664524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46962 (local address 127.0.97.2:44703)
0924 10:00:34.664707 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0924 10:00:34.664712 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:00:34.664731 (+ 19us) server_negotiation.cc:407] Connection header received
0924 10:00:34.664795 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:00:34.664801 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:00:34.664906 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:00:34.665008 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:00:34.665494 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:00:34.666722 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:00:34.668124 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:00:34.668785 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:00:34.668906 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:00:34.669842 (+ 936us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:00:34.669854 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:00:34.669872 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:00:34.670650 (+ 778us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:00:34.670709 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:00:34.670734 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:00:34.670834 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0924 10:00:34.670914 (+ 80us) server_negotiation.cc:299] Negotiation successful
0924 10:00:34.671014 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260924 10:00:34.672010 19475 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:00:34.719811 19658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:34.719300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43297 (local address 127.0.0.1:38768)
0924 10:00:34.719520 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0924 10:00:34.719628 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:43297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260924 10:00:34.720875 19658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:34.720445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43297 (local address 127.0.0.1:38774)
0924 10:00:34.720595 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0924 10:00:34.720696 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:43297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260924 10:00:34.781327 19658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:34.780807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43297 (local address 127.0.0.1:38784)
0924 10:00:34.781031 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0924 10:00:34.781155 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:43297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260924 10:00:34.782317 19658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:34.781941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43297 (local address 127.0.0.1:38800)
0924 10:00:34.782075 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0924 10:00:34.782154 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:43297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260924 10:00:34.784489 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu with pid 19353
I20260924 10:00:34.799165 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu pid 19483
I20260924 10:00:34.805625 19627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:34.799867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56966 (local address 127.0.97.3:42781)
0924 10:00:34.800014 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0924 10:00:34.800018 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:00:34.800034 (+ 16us) server_negotiation.cc:407] Connection header received
0924 10:00:34.800126 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:00:34.800130 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:00:34.800208 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:00:34.800287 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:00:34.800584 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:00:34.801278 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:00:34.802468 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:00:34.803003 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:00:34.803112 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:00:34.804186 (+ 1074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:00:34.804197 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:00:34.804217 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:00:34.804922 (+ 705us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:00:34.804980 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:00:34.805002 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:00:34.805098 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0924 10:00:34.805180 (+ 82us) server_negotiation.cc:299] Negotiation successful
0924 10:00:34.805284 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 10:00:34.806243 19605 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:00:34.862808 19658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:34.862238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43297 (local address 127.0.0.1:38808)
0924 10:00:34.862509 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0924 10:00:34.862622 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:43297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260924 10:00:34.863933 19658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:34.863485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43297 (local address 127.0.0.1:38812)
0924 10:00:34.863688 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0924 10:00:34.863779 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:43297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
I20260924 10:00:34.892653 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu with pid 19483
I20260924 10:00:34.905596 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu pid 19133
I20260924 10:00:34.906670 19194 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:00:35.010972 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu with pid 19133
2026-09-24T10:00:35Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (2910 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790243735634835-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:00:35 dist-test-slave-w8d2 krb5kdc[19664](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:00:35 dist-test-slave-w8d2 krb5kdc[19664](info): set up 2 sockets
Sep 24 10:00:35 dist-test-slave-w8d2 krb5kdc[19664](info): commencing operation
krb5kdc: starting...
W20260924 10:00:36.015439 388 mini_kdc.cc:121] Time spent starting KDC: real 0.968s user 0.001s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790243735634835-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790243735634835-388-0/krb5kdc/test-user.keytab.
Sep 24 10:00:36 dist-test-slave-w8d2 krb5kdc[19664](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244036, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:00:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:00:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790243735634835-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790243735634835-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260924 10:00:36.229817 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
/tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790243735634835-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790243735634835-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790243735634835-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790243735634835-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37431
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790243735634835-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39475
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37431
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790243735634835-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790243735634835-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790243735634835-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:00:36.540580 19677 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:00:36.540877 19677 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:00:36.541018 19677 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:00:36.548307 19677 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:00:36.548447 19677 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:00:36.548525 19677 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:00:36.548600 19677 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:00:36.548661 19677 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:00:36.558555 19677 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39475
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790243735634835-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790243735634835-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37431
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790243735634835-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37431
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790243735634835-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790243735634835-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:00:36.559930 19677 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:00:36.561277 19677 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:00:36.572772 19682 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:00:36.572857 19683 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:00:36.572815 19685 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:00:36.576738 19677 server_base.cc:1034] running on GCE node
I20260924 10:00:36.577514 19677 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:00:36.578938 19677 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:00:36.580130 19677 hybrid_clock.cc:648] HybridClock initialized: now 1790244036580089 us; error 28 us; skew 500 ppm
I20260924 10:00:36.580478 19677 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:00:36 dist-test-slave-w8d2 krb5kdc[19664](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244036, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:00:36.587561 19677 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260924 10:00:36.589133 19677 webserver.cc:460] Webserver started at http://127.0.97.62:33645/ using document root <none> and password file <none>
I20260924 10:00:36.589820 19677 fs_manager.cc:359] Metadata directory not provided
I20260924 10:00:36.589947 19677 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:00:36.590245 19677 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:00:36.593205 19677 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790243735634835-388-0/minicluster-data/master-0/data/instance:
uuid: "464c4899bba54f59af72bfebd7801b11"
format_stamp: "Formatted at 2026-09-24 10:00:36 on dist-test-slave-w8d2"
server_key: "486c049c28f51601893e35672e2f3128"
server_key_iv: "89e3c53598dd548e48f391dcf2efdaa8"
server_key_version: "encryptionkey@0"
I20260924 10:00:36.593945 19677 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790243735634835-388-0/minicluster-data/master-0/wal/instance:
uuid: "464c4899bba54f59af72bfebd7801b11"
format_stamp: "Formatted at 2026-09-24 10:00:36 on dist-test-slave-w8d2"
server_key: "486c049c28f51601893e35672e2f3128"
server_key_iv: "89e3c53598dd548e48f391dcf2efdaa8"
server_key_version: "encryptionkey@0"
I20260924 10:00:36.598857 19677 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260924 10:00:36.602226 19692 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:00:36.603554 19677 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 10:00:36.603701 19677 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790243735634835-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790243735634835-388-0/minicluster-data/master-0/wal
uuid: "464c4899bba54f59af72bfebd7801b11"
format_stamp: "Formatted at 2026-09-24 10:00:36 on dist-test-slave-w8d2"
server_key: "486c049c28f51601893e35672e2f3128"
server_key_iv: "89e3c53598dd548e48f391dcf2efdaa8"
server_key_version: "encryptionkey@0"
I20260924 10:00:36.603874 19677 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790243735634835-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790243735634835-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790243735634835-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:00:36.630246 19677 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:00:36.634476 19677 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:00:36.634730 19677 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:00:36.651871 19677 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:37431
I20260924 10:00:36.651907 19743 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:37431 every 8 connection(s)
I20260924 10:00:36.653235 19677 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790243735634835-388-0/minicluster-data/master-0/data/info.pb
I20260924 10:00:36.656924 19744 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:00:36.661265 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu as pid 19677
I20260924 10:00:36.661427 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790243735634835-388-0/minicluster-data/master-0/wal/instance
I20260924 10:00:36.661835 388 external_mini_cluster.cc:1428] Setting key 62462eb602df3c2ba3141f4d04051b02
I20260924 10:00:36.668295 19744 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 464c4899bba54f59af72bfebd7801b11: Bootstrap starting.
I20260924 10:00:36.671386 19744 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 464c4899bba54f59af72bfebd7801b11: Neither blocks nor log segments found. Creating new log.
I20260924 10:00:36.672449 19744 log.cc:824] T 00000000000000000000000000000000 P 464c4899bba54f59af72bfebd7801b11: Log is configured to *not* fsync() on all Append() calls
I20260924 10:00:36.675302 19744 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 464c4899bba54f59af72bfebd7801b11: No bootstrap required, opened a new log
Sep 24 10:00:36 dist-test-slave-w8d2 krb5kdc[19664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244036, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 10:00:36.683094 19744 raft_consensus.cc:348] T 00000000000000000000000000000000 P 464c4899bba54f59af72bfebd7801b11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "464c4899bba54f59af72bfebd7801b11" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37431 } }
I20260924 10:00:36.683554 19744 raft_consensus.cc:374] T 00000000000000000000000000000000 P 464c4899bba54f59af72bfebd7801b11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:00:36.683646 19744 raft_consensus.cc:729] T 00000000000000000000000000000000 P 464c4899bba54f59af72bfebd7801b11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 464c4899bba54f59af72bfebd7801b11, State: Initialized, Role: FOLLOWER
I20260924 10:00:36.683924 19747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:36.663637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54298 (local address 127.0.97.62:37431)
0924 10:00:36.664400 (+ 763us) server_negotiation.cc:206] Beginning negotiation
0924 10:00:36.664411 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 10:00:36.664455 (+ 44us) server_negotiation.cc:407] Connection header received
0924 10:00:36.666276 (+ 1821us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:00:36.666300 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:00:36.666738 (+ 438us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:00:36.667163 (+ 425us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:00:36.668937 (+ 1774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:00:36.670250 (+ 1313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:00:36.671553 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:00:36.672275 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:00:36.672461 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:00:36.676584 (+ 4123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:00:36.676592 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:00:36.676607 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:00:36.676651 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:00:36.679779 (+ 3128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:00:36.680452 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:00:36.680459 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:00:36.680466 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:00:36.680639 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:00:36.681009 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:00:36.681015 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:00:36.681017 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:00:36.681587 (+ 570us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:00:36.681890 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0924 10:00:36.682329 (+ 439us) server_negotiation.cc:299] Negotiation successful
0924 10:00:36.682745 (+ 416us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":549,"thread_start_us":349,"threads_started":1}
I20260924 10:00:36.684295 19744 consensus_queue.cc:260] T 00000000000000000000000000000000 P 464c4899bba54f59af72bfebd7801b11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "464c4899bba54f59af72bfebd7801b11" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37431 } }
I20260924 10:00:36.684571 19744 raft_consensus.cc:388] T 00000000000000000000000000000000 P 464c4899bba54f59af72bfebd7801b11 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:00:36.684705 19744 raft_consensus.cc:482] T 00000000000000000000000000000000 P 464c4899bba54f59af72bfebd7801b11 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:00:36.684882 19744 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 464c4899bba54f59af72bfebd7801b11 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:00:36.687395 19744 raft_consensus.cc:504] T 00000000000000000000000000000000 P 464c4899bba54f59af72bfebd7801b11 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "464c4899bba54f59af72bfebd7801b11" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37431 } }
I20260924 10:00:36.687824 19744 leader_election.cc:302] T 00000000000000000000000000000000 P 464c4899bba54f59af72bfebd7801b11 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 464c4899bba54f59af72bfebd7801b11; no voters:
I20260924 10:00:36.688315 19744 leader_election.cc:288] T 00000000000000000000000000000000 P 464c4899bba54f59af72bfebd7801b11 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:00:36.688436 19749 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 464c4899bba54f59af72bfebd7801b11 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:00:36.689095 19749 raft_consensus.cc:686] T 00000000000000000000000000000000 P 464c4899bba54f59af72bfebd7801b11 [term 1 LEADER]: Becoming Leader. State: Replica: 464c4899bba54f59af72bfebd7801b11, State: Running, Role: LEADER
I20260924 10:00:36.689594 19749 consensus_queue.cc:237] T 00000000000000000000000000000000 P 464c4899bba54f59af72bfebd7801b11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "464c4899bba54f59af72bfebd7801b11" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37431 } }
I20260924 10:00:36.690253 19744 sys_catalog.cc:564] T 00000000000000000000000000000000 P 464c4899bba54f59af72bfebd7801b11 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:00:36.692049 19750 sys_catalog.cc:455] T 00000000000000000000000000000000 P 464c4899bba54f59af72bfebd7801b11 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "464c4899bba54f59af72bfebd7801b11" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "464c4899bba54f59af72bfebd7801b11" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37431 } } }
I20260924 10:00:36.692423 19750 sys_catalog.cc:458] T 00000000000000000000000000000000 P 464c4899bba54f59af72bfebd7801b11 [sys.catalog]: This master's current role is: LEADER
I20260924 10:00:36.692924 19751 sys_catalog.cc:455] T 00000000000000000000000000000000 P 464c4899bba54f59af72bfebd7801b11 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 464c4899bba54f59af72bfebd7801b11. Latest consensus state: current_term: 1 leader_uuid: "464c4899bba54f59af72bfebd7801b11" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "464c4899bba54f59af72bfebd7801b11" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37431 } } }
I20260924 10:00:36.693221 19751 sys_catalog.cc:458] T 00000000000000000000000000000000 P 464c4899bba54f59af72bfebd7801b11 [sys.catalog]: This master's current role is: LEADER
I20260924 10:00:36.694072 19754 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:00:36.699072 19754 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:00:36.708506 19754 catalog_manager.cc:1347] Generated new cluster ID: 499f691303634100b991549f0300e27a
I20260924 10:00:36.708629 19754 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:00:36.720456 19754 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:00:36.721828 19754 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:00:36.729950 19754 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 464c4899bba54f59af72bfebd7801b11: Generated new TSK 0
I20260924 10:00:36.730736 19754 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790243735634835-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790243735634835-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260924 10:00:36.788676 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
/tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790243735634835-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790243735634835-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790243735634835-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790243735634835-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37431
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790243735634835-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39475
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790243735634835-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790243735634835-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790243735634835-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:00:37.107048 19770 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:00:37.107402 19770 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:00:37.107543 19770 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:00:37.114929 19770 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:00:37.115168 19770 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:00:37.115372 19770 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260924 10:00:37.125155 19770 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39475
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790243735634835-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790243735634835-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790243735634835-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790243735634835-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37431
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790243735634835-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:00:37.126418 19770 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:00:37.127691 19770 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:00:37.139909 19775 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:00:37.139871 19778 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:00:37.139883 19776 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:00:37.140717 19770 server_base.cc:1034] running on GCE node
I20260924 10:00:37.141310 19770 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:00:37.142210 19770 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:00:37.143409 19770 hybrid_clock.cc:648] HybridClock initialized: now 1790244037143346 us; error 53 us; skew 500 ppm
I20260924 10:00:37.143715 19770 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:00:37 dist-test-slave-w8d2 krb5kdc[19664](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244037, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:00:37.146674 19770 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260924 10:00:37.148085 19770 webserver.cc:460] Webserver started at http://127.0.97.1:41277/ using document root <none> and password file <none>
I20260924 10:00:37.148722 19770 fs_manager.cc:359] Metadata directory not provided
I20260924 10:00:37.148831 19770 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:00:37.149120 19770 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:00:37.151889 19770 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790243735634835-388-0/minicluster-data/ts-0/data/instance:
uuid: "8a8c6d9364c04579bbbf3a180ca87f9b"
format_stamp: "Formatted at 2026-09-24 10:00:37 on dist-test-slave-w8d2"
server_key: "f8d21bbf7c63842f32d0cbe9645e69eb"
server_key_iv: "ea285ff1433f108de8f3a535347a41a3"
server_key_version: "encryptionkey@0"
I20260924 10:00:37.152642 19770 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790243735634835-388-0/minicluster-data/ts-0/wal/instance:
uuid: "8a8c6d9364c04579bbbf3a180ca87f9b"
format_stamp: "Formatted at 2026-09-24 10:00:37 on dist-test-slave-w8d2"
server_key: "f8d21bbf7c63842f32d0cbe9645e69eb"
server_key_iv: "ea285ff1433f108de8f3a535347a41a3"
server_key_version: "encryptionkey@0"
I20260924 10:00:37.157469 19770 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 10:00:37.160773 19785 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:00:37.162047 19770 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 10:00:37.162182 19770 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790243735634835-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790243735634835-388-0/minicluster-data/ts-0/wal
uuid: "8a8c6d9364c04579bbbf3a180ca87f9b"
format_stamp: "Formatted at 2026-09-24 10:00:37 on dist-test-slave-w8d2"
server_key: "f8d21bbf7c63842f32d0cbe9645e69eb"
server_key_iv: "ea285ff1433f108de8f3a535347a41a3"
server_key_version: "encryptionkey@0"
I20260924 10:00:37.162360 19770 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790243735634835-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790243735634835-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790243735634835-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:00:37.177268 19770 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:00:37.181066 19770 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:00:37.181349 19770 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:00:37.182394 19770 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:00:37.183776 19770 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:00:37.183849 19770 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:00:37.183971 19770 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:00:37.184029 19770 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:00:37.218449 19770 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:40969
I20260924 10:00:37.218492 19897 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:40969 every 8 connection(s)
I20260924 10:00:37.219772 19770 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790243735634835-388-0/minicluster-data/ts-0/data/info.pb
I20260924 10:00:37.222432 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu as pid 19770
I20260924 10:00:37.222604 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790243735634835-388-0/minicluster-data/ts-0/wal/instance
I20260924 10:00:37.223006 388 external_mini_cluster.cc:1428] Setting key d2f831955649ae0518fae1c34e7443c1
Sep 24 10:00:37 dist-test-slave-w8d2 krb5kdc[19664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244037, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 10:00:37.239068 19900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:37.222189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:48879 (local address 127.0.97.62:37431)
0924 10:00:37.222680 (+ 491us) server_negotiation.cc:206] Beginning negotiation
0924 10:00:37.222686 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:00:37.223907 (+ 1221us) server_negotiation.cc:407] Connection header received
0924 10:00:37.225503 (+ 1596us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:00:37.225508 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:00:37.225589 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:00:37.225680 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:00:37.227143 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:00:37.227903 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:00:37.229535 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:00:37.229959 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:00:37.230091 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:00:37.233698 (+ 3607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:00:37.233703 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:00:37.233707 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:00:37.233740 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:00:37.236181 (+ 2441us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:00:37.236862 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:00:37.236868 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:00:37.236871 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:00:37.236985 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:00:37.237383 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:00:37.237388 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:00:37.237392 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:00:37.237656 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:00:37.237793 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0924 10:00:37.238473 (+ 680us) server_negotiation.cc:299] Negotiation successful
0924 10:00:37.238702 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":225,"threads_started":1}
I20260924 10:00:37.240182 19901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:37.222719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37431 (local address 127.0.97.1:48879)
0924 10:00:37.223532 (+ 813us) negotiation.cc:107] Waiting for socket to connect
0924 10:00:37.223584 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0924 10:00:37.225242 (+ 1658us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:00:37.225923 (+ 681us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:00:37.225931 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:00:37.226415 (+ 484us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:00:37.226983 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:00:37.227001 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:00:37.228050 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:00:37.228054 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:00:37.229403 (+ 1349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:00:37.229415 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:00:37.230184 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:00:37.230191 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:00:37.230418 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:00:37.231089 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:00:37.231113 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:00:37.233472 (+ 2359us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:00:37.236312 (+ 2840us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:00:37.236321 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:00:37.236339 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:00:37.236752 (+ 413us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:00:37.237085 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:00:37.237090 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:00:37.237094 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:00:37.237281 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:00:37.237780 (+ 499us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:00:37.237789 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:00:37.238130 (+ 341us) client_negotiation.cc:769] Sending connection context
0924 10:00:37.238374 (+ 244us) client_negotiation.cc:240] Negotiation successful
0924 10:00:37.238917 (+ 543us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":541,"thread_start_us":341,"threads_started":1}
I20260924 10:00:37.241775 19898 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37431
I20260924 10:00:37.242132 19898 heartbeater.cc:464] Registering TS with master...
I20260924 10:00:37.243054 19898 heartbeater.cc:511] Master 127.0.97.62:37431 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260924 10:00:37.245306 19709 ts_manager.cc:194] Registered new tserver with Master: 8a8c6d9364c04579bbbf3a180ca87f9b (127.0.97.1:40969)
I20260924 10:00:37.246707 19709 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:48879
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790243735634835-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790243735634835-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260924 10:00:37.273481 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
/tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790243735634835-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790243735634835-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790243735634835-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790243735634835-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37431
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790243735634835-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39475
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790243735634835-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790243735634835-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790243735634835-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:00:37.579668 19904 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:00:37.579979 19904 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:00:37.580077 19904 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:00:37.587272 19904 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:00:37.587394 19904 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:00:37.587517 19904 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260924 10:00:37.597348 19904 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39475
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790243735634835-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790243735634835-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790243735634835-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790243735634835-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37431
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790243735634835-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:00:37.598582 19904 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:00:37.599892 19904 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:00:37.612152 19909 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:00:37.612098 19910 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:00:37.612123 19912 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:00:37.613039 19904 server_base.cc:1034] running on GCE node
I20260924 10:00:37.613579 19904 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:00:37.614428 19904 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:00:37.615639 19904 hybrid_clock.cc:648] HybridClock initialized: now 1790244037615587 us; error 38 us; skew 500 ppm
I20260924 10:00:37.615931 19904 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:00:37 dist-test-slave-w8d2 krb5kdc[19664](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244037, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:00:37.618845 19904 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260924 10:00:37.620204 19904 webserver.cc:460] Webserver started at http://127.0.97.2:42375/ using document root <none> and password file <none>
I20260924 10:00:37.620855 19904 fs_manager.cc:359] Metadata directory not provided
I20260924 10:00:37.620963 19904 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:00:37.621232 19904 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:00:37.624086 19904 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790243735634835-388-0/minicluster-data/ts-1/data/instance:
uuid: "924d4d9496ef42f0a704dec2a36af1b5"
format_stamp: "Formatted at 2026-09-24 10:00:37 on dist-test-slave-w8d2"
server_key: "23f96420cb357f2a102e1389e3fe513b"
server_key_iv: "e456d09d1fd63d88bd8c018a3ec46232"
server_key_version: "encryptionkey@0"
I20260924 10:00:37.624805 19904 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790243735634835-388-0/minicluster-data/ts-1/wal/instance:
uuid: "924d4d9496ef42f0a704dec2a36af1b5"
format_stamp: "Formatted at 2026-09-24 10:00:37 on dist-test-slave-w8d2"
server_key: "23f96420cb357f2a102e1389e3fe513b"
server_key_iv: "e456d09d1fd63d88bd8c018a3ec46232"
server_key_version: "encryptionkey@0"
I20260924 10:00:37.629874 19904 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 10:00:37.633249 19919 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:00:37.634471 19904 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 10:00:37.634613 19904 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790243735634835-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790243735634835-388-0/minicluster-data/ts-1/wal
uuid: "924d4d9496ef42f0a704dec2a36af1b5"
format_stamp: "Formatted at 2026-09-24 10:00:37 on dist-test-slave-w8d2"
server_key: "23f96420cb357f2a102e1389e3fe513b"
server_key_iv: "e456d09d1fd63d88bd8c018a3ec46232"
server_key_version: "encryptionkey@0"
I20260924 10:00:37.634806 19904 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790243735634835-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790243735634835-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790243735634835-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:00:37.669042 19904 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:00:37.673434 19904 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:00:37.673686 19904 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:00:37.674633 19904 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:00:37.676023 19904 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:00:37.676098 19904 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:00:37.676220 19904 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:00:37.676273 19904 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:00:37.705682 19904 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:38307
I20260924 10:00:37.705737 20031 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:38307 every 8 connection(s)
I20260924 10:00:37.706907 19904 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790243735634835-388-0/minicluster-data/ts-1/data/info.pb
I20260924 10:00:37.714032 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu as pid 19904
I20260924 10:00:37.714183 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790243735634835-388-0/minicluster-data/ts-1/wal/instance
I20260924 10:00:37.714599 388 external_mini_cluster.cc:1428] Setting key 09d34e0ae11f55003a0439a3c9d47b11
Sep 24 10:00:37 dist-test-slave-w8d2 krb5kdc[19664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244037, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 10:00:37.726430 19900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:37.709421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:36863 (local address 127.0.97.62:37431)
0924 10:00:37.709604 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0924 10:00:37.709609 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:00:37.710791 (+ 1182us) server_negotiation.cc:407] Connection header received
0924 10:00:37.712614 (+ 1823us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:00:37.712619 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:00:37.712687 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:00:37.712762 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:00:37.714222 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:00:37.715005 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:00:37.716796 (+ 1791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:00:37.717202 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:00:37.717309 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:00:37.720700 (+ 3391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:00:37.720706 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:00:37.720710 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:00:37.720748 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:00:37.723646 (+ 2898us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:00:37.724362 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:00:37.724366 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:00:37.724368 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:00:37.724490 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:00:37.724932 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:00:37.724935 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:00:37.724937 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:00:37.725182 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:00:37.725296 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0924 10:00:37.726034 (+ 738us) server_negotiation.cc:299] Negotiation successful
0924 10:00:37.726201 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260924 10:00:37.727769 20034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:37.709911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37431 (local address 127.0.97.2:36863)
0924 10:00:37.710629 (+ 718us) negotiation.cc:107] Waiting for socket to connect
0924 10:00:37.710665 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 10:00:37.712363 (+ 1698us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:00:37.712956 (+ 593us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:00:37.712963 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:00:37.713476 (+ 513us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:00:37.714009 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:00:37.714026 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:00:37.715139 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:00:37.715143 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:00:37.716661 (+ 1518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:00:37.716673 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:00:37.717421 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:00:37.717428 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:00:37.717641 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:00:37.718238 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:00:37.718265 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:00:37.720471 (+ 2206us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:00:37.723761 (+ 3290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:00:37.723767 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:00:37.723780 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:00:37.724248 (+ 468us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:00:37.724577 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:00:37.724582 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:00:37.724585 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:00:37.724818 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:00:37.725295 (+ 477us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:00:37.725303 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:00:37.725678 (+ 375us) client_negotiation.cc:769] Sending connection context
0924 10:00:37.725969 (+ 291us) client_negotiation.cc:240] Negotiation successful
0924 10:00:37.726459 (+ 490us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":302,"threads_started":1}
I20260924 10:00:37.729105 20032 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37431
I20260924 10:00:37.729418 20032 heartbeater.cc:464] Registering TS with master...
I20260924 10:00:37.730182 20032 heartbeater.cc:511] Master 127.0.97.62:37431 requested a full tablet report, sending...
I20260924 10:00:37.731706 19709 ts_manager.cc:194] Registered new tserver with Master: 924d4d9496ef42f0a704dec2a36af1b5 (127.0.97.2:38307)
I20260924 10:00:37.732692 19709 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:36863
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790243735634835-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790243735634835-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260924 10:00:37.764356 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
/tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790243735634835-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790243735634835-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790243735634835-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790243735634835-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37431
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790243735634835-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39475
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790243735634835-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790243735634835-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790243735634835-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:00:38.071913 20037 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:00:38.072178 20037 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:00:38.072278 20037 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:00:38.079464 20037 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:00:38.079559 20037 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:00:38.079674 20037 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260924 10:00:38.088999 20037 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39475
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790243735634835-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790243735634835-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790243735634835-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790243735634835-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37431
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790243735634835-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:00:38.090149 20037 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:00:38.091387 20037 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:00:38.103894 20042 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:00:38.103927 20045 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:00:38.103960 20043 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:00:38.104650 20037 server_base.cc:1034] running on GCE node
I20260924 10:00:38.105189 20037 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:00:38.106026 20037 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:00:38.107226 20037 hybrid_clock.cc:648] HybridClock initialized: now 1790244038107171 us; error 34 us; skew 500 ppm
I20260924 10:00:38.107553 20037 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:00:38 dist-test-slave-w8d2 krb5kdc[19664](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244038, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:00:38.110395 20037 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260924 10:00:38.111836 20037 webserver.cc:460] Webserver started at http://127.0.97.3:43147/ using document root <none> and password file <none>
I20260924 10:00:38.112496 20037 fs_manager.cc:359] Metadata directory not provided
I20260924 10:00:38.112607 20037 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:00:38.112890 20037 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:00:38.115854 20037 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790243735634835-388-0/minicluster-data/ts-2/data/instance:
uuid: "ee93d05f71604a54a90ca3d60e798c0c"
format_stamp: "Formatted at 2026-09-24 10:00:38 on dist-test-slave-w8d2"
server_key: "2e32b70c8b6ffab3e5ff6d4dedc8543e"
server_key_iv: "030b732a14c5d611fd7e65a37a36c0b5"
server_key_version: "encryptionkey@0"
I20260924 10:00:38.116618 20037 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790243735634835-388-0/minicluster-data/ts-2/wal/instance:
uuid: "ee93d05f71604a54a90ca3d60e798c0c"
format_stamp: "Formatted at 2026-09-24 10:00:38 on dist-test-slave-w8d2"
server_key: "2e32b70c8b6ffab3e5ff6d4dedc8543e"
server_key_iv: "030b732a14c5d611fd7e65a37a36c0b5"
server_key_version: "encryptionkey@0"
I20260924 10:00:38.121495 20037 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260924 10:00:38.124892 20052 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:00:38.126185 20037 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 10:00:38.126331 20037 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790243735634835-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790243735634835-388-0/minicluster-data/ts-2/wal
uuid: "ee93d05f71604a54a90ca3d60e798c0c"
format_stamp: "Formatted at 2026-09-24 10:00:38 on dist-test-slave-w8d2"
server_key: "2e32b70c8b6ffab3e5ff6d4dedc8543e"
server_key_iv: "030b732a14c5d611fd7e65a37a36c0b5"
server_key_version: "encryptionkey@0"
I20260924 10:00:38.126504 20037 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790243735634835-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790243735634835-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790243735634835-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:00:38.144929 20037 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:00:38.149233 20037 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:00:38.149513 20037 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:00:38.150477 20037 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:00:38.151876 20037 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:00:38.151957 20037 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:00:38.152091 20037 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:00:38.152143 20037 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:00:38.181911 20037 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:37505
I20260924 10:00:38.181950 20164 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:37505 every 8 connection(s)
I20260924 10:00:38.183152 20037 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790243735634835-388-0/minicluster-data/ts-2/data/info.pb
I20260924 10:00:38.185221 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu as pid 20037
I20260924 10:00:38.185364 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790243735634835-388-0/minicluster-data/ts-2/wal/instance
I20260924 10:00:38.185849 388 external_mini_cluster.cc:1428] Setting key 04189d26a145d099cfd54767c7e27e14
Sep 24 10:00:38 dist-test-slave-w8d2 krb5kdc[19664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244038, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 10:00:38.202304 19900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:38.185621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:42565 (local address 127.0.97.62:37431)
0924 10:00:38.185823 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0924 10:00:38.185830 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:00:38.187002 (+ 1172us) server_negotiation.cc:407] Connection header received
0924 10:00:38.189218 (+ 2216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:00:38.189224 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:00:38.189304 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:00:38.189385 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:00:38.190991 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:00:38.191813 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:00:38.193302 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:00:38.193681 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:00:38.193787 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:00:38.197171 (+ 3384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:00:38.197176 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:00:38.197179 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:00:38.197212 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:00:38.199483 (+ 2271us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:00:38.200144 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:00:38.200149 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:00:38.200153 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:00:38.200279 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:00:38.200668 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:00:38.200673 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:00:38.200676 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:00:38.200945 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:00:38.201071 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0924 10:00:38.201744 (+ 673us) server_negotiation.cc:299] Negotiation successful
0924 10:00:38.201959 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260924 10:00:38.203264 20167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:38.186104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37431 (local address 127.0.97.3:42565)
0924 10:00:38.186851 (+ 747us) negotiation.cc:107] Waiting for socket to connect
0924 10:00:38.186902 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0924 10:00:38.188965 (+ 2063us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:00:38.189635 (+ 670us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:00:38.189643 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:00:38.190184 (+ 541us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:00:38.190815 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:00:38.190833 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:00:38.191962 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:00:38.191966 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:00:38.193170 (+ 1204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:00:38.193182 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:00:38.193896 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:00:38.193903 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:00:38.194129 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:00:38.194720 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:00:38.194748 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:00:38.196949 (+ 2201us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:00:38.199606 (+ 2657us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:00:38.199616 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:00:38.199633 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:00:38.200035 (+ 402us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:00:38.200377 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:00:38.200382 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:00:38.200386 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:00:38.200566 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:00:38.201063 (+ 497us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:00:38.201070 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:00:38.201396 (+ 326us) client_negotiation.cc:769] Sending connection context
0924 10:00:38.201641 (+ 245us) client_negotiation.cc:240] Negotiation successful
0924 10:00:38.202177 (+ 536us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":278,"threads_started":1}
I20260924 10:00:38.204501 20165 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37431
I20260924 10:00:38.204799 20165 heartbeater.cc:464] Registering TS with master...
I20260924 10:00:38.205489 20165 heartbeater.cc:511] Master 127.0.97.62:37431 requested a full tablet report, sending...
I20260924 10:00:38.206830 19709 ts_manager.cc:194] Registered new tserver with Master: ee93d05f71604a54a90ca3d60e798c0c (127.0.97.3:37505)
I20260924 10:00:38.207751 19709 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:42565
I20260924 10:00:38.212821 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 10:00:38.228523 19900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:38.216565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54308 (local address 127.0.97.62:37431)
0924 10:00:38.216776 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0924 10:00:38.216783 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:00:38.217088 (+ 305us) server_negotiation.cc:407] Connection header received
0924 10:00:38.217217 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:00:38.217221 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:00:38.217281 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:00:38.217356 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:00:38.218791 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:00:38.219917 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:00:38.221131 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:00:38.221748 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:00:38.221862 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:00:38.224125 (+ 2263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:00:38.224130 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:00:38.224133 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:00:38.224177 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:00:38.226396 (+ 2219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:00:38.226974 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:00:38.226977 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:00:38.226979 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:00:38.227083 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:00:38.227398 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:00:38.227401 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:00:38.227403 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:00:38.227691 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:00:38.227867 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0924 10:00:38.228135 (+ 268us) server_negotiation.cc:299] Negotiation successful
0924 10:00:38.228306 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 10:00:38.232290 19709 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54308:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 10:00:38.239120 19709 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 10:00:38.249421 19898 heartbeater.cc:503] Master 127.0.97.62:37431 was elected leader, sending a full tablet report...
I20260924 10:00:38.273092 20182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:38.252318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45858 (local address 127.0.97.1:40969)
0924 10:00:38.263779 (+ 11461us) server_negotiation.cc:206] Beginning negotiation
0924 10:00:38.263788 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 10:00:38.264424 (+ 636us) server_negotiation.cc:407] Connection header received
0924 10:00:38.264532 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:00:38.264540 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:00:38.264848 (+ 308us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:00:38.265035 (+ 187us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:00:38.265786 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:00:38.266934 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:00:38.270227 (+ 3293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:00:38.271467 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:00:38.271589 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:00:38.272185 (+ 596us) server_negotiation.cc:1035] Waiting for connection context
0924 10:00:38.272717 (+ 532us) server_negotiation.cc:299] Negotiation successful
0924 10:00:38.272857 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11287,"thread_start_us":11122,"threads_started":1}
I20260924 10:00:38.273196 20181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:38.251854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40969 (local address 127.0.0.1:45858)
0924 10:00:38.264164 (+ 12310us) negotiation.cc:107] Waiting for socket to connect
0924 10:00:38.264186 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 10:00:38.264305 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:00:38.265332 (+ 1027us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:00:38.265337 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:00:38.265361 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:00:38.265615 (+ 254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:00:38.265622 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:00:38.267617 (+ 1995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:00:38.267623 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:00:38.270052 (+ 2429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:00:38.270070 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:00:38.272302 (+ 2232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:00:38.272307 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:00:38.272470 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:00:38.272525 (+ 55us) client_negotiation.cc:769] Sending connection context
0924 10:00:38.272654 (+ 129us) client_negotiation.cc:240] Negotiation successful
0924 10:00:38.272848 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12177,"thread_start_us":12192,"threads_started":1}
I20260924 10:00:38.274887 20177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:38.250532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:38307 (local address 127.0.0.1:57240)
0924 10:00:38.256920 (+ 6388us) negotiation.cc:107] Waiting for socket to connect
0924 10:00:38.256973 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0924 10:00:38.257164 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:00:38.267577 (+ 10413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:00:38.267582 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:00:38.267609 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:00:38.268081 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:00:38.268093 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:00:38.270134 (+ 2041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:00:38.270139 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:00:38.272360 (+ 2221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:00:38.272370 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:00:38.274120 (+ 1750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:00:38.274126 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:00:38.274285 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:00:38.274316 (+ 31us) client_negotiation.cc:769] Sending connection context
0924 10:00:38.274411 (+ 95us) client_negotiation.cc:240] Negotiation successful
0924 10:00:38.274532 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6116,"mutex_wait_us":30,"thread_start_us":465,"threads_started":1}
I20260924 10:00:38.274931 20178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:38.250698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57240 (local address 127.0.97.2:38307)
0924 10:00:38.266901 (+ 16203us) server_negotiation.cc:206] Beginning negotiation
0924 10:00:38.266912 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 10:00:38.266946 (+ 34us) server_negotiation.cc:407] Connection header received
0924 10:00:38.267050 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:00:38.267060 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:00:38.267344 (+ 284us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:00:38.268289 (+ 945us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:00:38.268507 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:00:38.269999 (+ 1492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:00:38.272512 (+ 2513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:00:38.273928 (+ 1416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:00:38.274048 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:00:38.274235 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0924 10:00:38.274453 (+ 218us) server_negotiation.cc:299] Negotiation successful
0924 10:00:38.274620 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16002,"thread_start_us":16712,"threads_started":1}
I20260924 10:00:38.276300 20179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:38.250916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:37505 (local address 127.0.0.1:48144)
0924 10:00:38.263090 (+ 12174us) negotiation.cc:107] Waiting for socket to connect
0924 10:00:38.263117 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0924 10:00:38.263293 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:00:38.268095 (+ 4802us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:00:38.268099 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:00:38.268126 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:00:38.268496 (+ 370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:00:38.268506 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:00:38.269930 (+ 1424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:00:38.269936 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:00:38.272333 (+ 2397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:00:38.272352 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:00:38.275542 (+ 3190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:00:38.275548 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:00:38.275715 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:00:38.275746 (+ 31us) client_negotiation.cc:769] Sending connection context
0924 10:00:38.275827 (+ 81us) client_negotiation.cc:240] Negotiation successful
0924 10:00:38.275946 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11950,"mutex_wait_us":87,"thread_start_us":11775,"threads_started":1}
I20260924 10:00:38.277067 20180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:38.251543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48144 (local address 127.0.97.3:37505)
0924 10:00:38.267530 (+ 15987us) server_negotiation.cc:206] Beginning negotiation
0924 10:00:38.267539 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 10:00:38.267572 (+ 33us) server_negotiation.cc:407] Connection header received
0924 10:00:38.267672 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:00:38.267678 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:00:38.267896 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:00:38.268661 (+ 765us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:00:38.268851 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:00:38.269791 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:00:38.273693 (+ 3902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:00:38.275034 (+ 1341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:00:38.275151 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:00:38.275390 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0924 10:00:38.276580 (+ 1190us) server_negotiation.cc:299] Negotiation successful
0924 10:00:38.276748 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15817,"thread_start_us":15624,"threads_started":1}
I20260924 10:00:38.277068 19833 tablet_service.cc:1467] Processing CreateTablet for tablet b310367497754ac5b7132de2ef85f477 (DEFAULT_TABLE table=test-table [id=c97b3aae7e974e94a3d232c46510184f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:00:38.278443 19833 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b310367497754ac5b7132de2ef85f477. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:00:38.278614 19967 tablet_service.cc:1467] Processing CreateTablet for tablet b310367497754ac5b7132de2ef85f477 (DEFAULT_TABLE table=test-table [id=c97b3aae7e974e94a3d232c46510184f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:00:38.279682 19967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b310367497754ac5b7132de2ef85f477. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:00:38.280308 20100 tablet_service.cc:1467] Processing CreateTablet for tablet b310367497754ac5b7132de2ef85f477 (DEFAULT_TABLE table=test-table [id=c97b3aae7e974e94a3d232c46510184f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:00:38.281554 20100 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b310367497754ac5b7132de2ef85f477. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:00:38.290199 20183 tablet_bootstrap.cc:492] T b310367497754ac5b7132de2ef85f477 P 924d4d9496ef42f0a704dec2a36af1b5: Bootstrap starting.
I20260924 10:00:38.293499 20183 tablet_bootstrap.cc:654] T b310367497754ac5b7132de2ef85f477 P 924d4d9496ef42f0a704dec2a36af1b5: Neither blocks nor log segments found. Creating new log.
I20260924 10:00:38.294911 20184 tablet_bootstrap.cc:492] T b310367497754ac5b7132de2ef85f477 P 8a8c6d9364c04579bbbf3a180ca87f9b: Bootstrap starting.
I20260924 10:00:38.294911 20183 log.cc:824] T b310367497754ac5b7132de2ef85f477 P 924d4d9496ef42f0a704dec2a36af1b5: Log is configured to *not* fsync() on all Append() calls
I20260924 10:00:38.296408 20185 tablet_bootstrap.cc:492] T b310367497754ac5b7132de2ef85f477 P ee93d05f71604a54a90ca3d60e798c0c: Bootstrap starting.
I20260924 10:00:38.298795 20184 tablet_bootstrap.cc:654] T b310367497754ac5b7132de2ef85f477 P 8a8c6d9364c04579bbbf3a180ca87f9b: Neither blocks nor log segments found. Creating new log.
I20260924 10:00:38.299360 20185 tablet_bootstrap.cc:654] T b310367497754ac5b7132de2ef85f477 P ee93d05f71604a54a90ca3d60e798c0c: Neither blocks nor log segments found. Creating new log.
I20260924 10:00:38.299957 20184 log.cc:824] T b310367497754ac5b7132de2ef85f477 P 8a8c6d9364c04579bbbf3a180ca87f9b: Log is configured to *not* fsync() on all Append() calls
I20260924 10:00:38.300444 20185 log.cc:824] T b310367497754ac5b7132de2ef85f477 P ee93d05f71604a54a90ca3d60e798c0c: Log is configured to *not* fsync() on all Append() calls
I20260924 10:00:38.302000 20183 tablet_bootstrap.cc:492] T b310367497754ac5b7132de2ef85f477 P 924d4d9496ef42f0a704dec2a36af1b5: No bootstrap required, opened a new log
I20260924 10:00:38.302328 20183 ts_tablet_manager.cc:1397] T b310367497754ac5b7132de2ef85f477 P 924d4d9496ef42f0a704dec2a36af1b5: Time spent bootstrapping tablet: real 0.012s user 0.011s sys 0.000s
I20260924 10:00:38.302616 20184 tablet_bootstrap.cc:492] T b310367497754ac5b7132de2ef85f477 P 8a8c6d9364c04579bbbf3a180ca87f9b: No bootstrap required, opened a new log
I20260924 10:00:38.302893 20184 ts_tablet_manager.cc:1397] T b310367497754ac5b7132de2ef85f477 P 8a8c6d9364c04579bbbf3a180ca87f9b: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.006s
I20260924 10:00:38.303258 20185 tablet_bootstrap.cc:492] T b310367497754ac5b7132de2ef85f477 P ee93d05f71604a54a90ca3d60e798c0c: No bootstrap required, opened a new log
I20260924 10:00:38.303498 20185 ts_tablet_manager.cc:1397] T b310367497754ac5b7132de2ef85f477 P ee93d05f71604a54a90ca3d60e798c0c: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.007s
I20260924 10:00:38.309852 20183 raft_consensus.cc:348] T b310367497754ac5b7132de2ef85f477 P 924d4d9496ef42f0a704dec2a36af1b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "924d4d9496ef42f0a704dec2a36af1b5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38307 } } peers { permanent_uuid: "ee93d05f71604a54a90ca3d60e798c0c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37505 } } peers { permanent_uuid: "8a8c6d9364c04579bbbf3a180ca87f9b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40969 } }
I20260924 10:00:38.310248 20183 raft_consensus.cc:374] T b310367497754ac5b7132de2ef85f477 P 924d4d9496ef42f0a704dec2a36af1b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:00:38.310339 20183 raft_consensus.cc:729] T b310367497754ac5b7132de2ef85f477 P 924d4d9496ef42f0a704dec2a36af1b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 924d4d9496ef42f0a704dec2a36af1b5, State: Initialized, Role: FOLLOWER
I20260924 10:00:38.310932 20183 consensus_queue.cc:260] T b310367497754ac5b7132de2ef85f477 P 924d4d9496ef42f0a704dec2a36af1b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "924d4d9496ef42f0a704dec2a36af1b5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38307 } } peers { permanent_uuid: "ee93d05f71604a54a90ca3d60e798c0c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37505 } } peers { permanent_uuid: "8a8c6d9364c04579bbbf3a180ca87f9b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40969 } }
I20260924 10:00:38.312028 20032 heartbeater.cc:503] Master 127.0.97.62:37431 was elected leader, sending a full tablet report...
I20260924 10:00:38.312743 20183 ts_tablet_manager.cc:1428] T b310367497754ac5b7132de2ef85f477 P 924d4d9496ef42f0a704dec2a36af1b5: Time spent starting tablet: real 0.010s user 0.009s sys 0.000s
I20260924 10:00:38.313362 20184 raft_consensus.cc:348] T b310367497754ac5b7132de2ef85f477 P 8a8c6d9364c04579bbbf3a180ca87f9b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "924d4d9496ef42f0a704dec2a36af1b5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38307 } } peers { permanent_uuid: "ee93d05f71604a54a90ca3d60e798c0c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37505 } } peers { permanent_uuid: "8a8c6d9364c04579bbbf3a180ca87f9b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40969 } }
I20260924 10:00:38.313493 20185 raft_consensus.cc:348] T b310367497754ac5b7132de2ef85f477 P ee93d05f71604a54a90ca3d60e798c0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "924d4d9496ef42f0a704dec2a36af1b5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38307 } } peers { permanent_uuid: "ee93d05f71604a54a90ca3d60e798c0c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37505 } } peers { permanent_uuid: "8a8c6d9364c04579bbbf3a180ca87f9b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40969 } }
I20260924 10:00:38.313836 20184 raft_consensus.cc:374] T b310367497754ac5b7132de2ef85f477 P 8a8c6d9364c04579bbbf3a180ca87f9b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:00:38.313910 20185 raft_consensus.cc:374] T b310367497754ac5b7132de2ef85f477 P ee93d05f71604a54a90ca3d60e798c0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:00:38.313933 20184 raft_consensus.cc:729] T b310367497754ac5b7132de2ef85f477 P 8a8c6d9364c04579bbbf3a180ca87f9b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a8c6d9364c04579bbbf3a180ca87f9b, State: Initialized, Role: FOLLOWER
I20260924 10:00:38.313987 20185 raft_consensus.cc:729] T b310367497754ac5b7132de2ef85f477 P ee93d05f71604a54a90ca3d60e798c0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee93d05f71604a54a90ca3d60e798c0c, State: Initialized, Role: FOLLOWER
I20260924 10:00:38.314651 20184 consensus_queue.cc:260] T b310367497754ac5b7132de2ef85f477 P 8a8c6d9364c04579bbbf3a180ca87f9b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "924d4d9496ef42f0a704dec2a36af1b5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38307 } } peers { permanent_uuid: "ee93d05f71604a54a90ca3d60e798c0c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37505 } } peers { permanent_uuid: "8a8c6d9364c04579bbbf3a180ca87f9b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40969 } }
I20260924 10:00:38.314651 20185 consensus_queue.cc:260] T b310367497754ac5b7132de2ef85f477 P ee93d05f71604a54a90ca3d60e798c0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "924d4d9496ef42f0a704dec2a36af1b5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38307 } } peers { permanent_uuid: "ee93d05f71604a54a90ca3d60e798c0c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37505 } } peers { permanent_uuid: "8a8c6d9364c04579bbbf3a180ca87f9b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40969 } }
I20260924 10:00:38.316259 20165 heartbeater.cc:503] Master 127.0.97.62:37431 was elected leader, sending a full tablet report...
I20260924 10:00:38.316469 20184 ts_tablet_manager.cc:1428] T b310367497754ac5b7132de2ef85f477 P 8a8c6d9364c04579bbbf3a180ca87f9b: Time spent starting tablet: real 0.013s user 0.012s sys 0.002s
I20260924 10:00:38.316469 20185 ts_tablet_manager.cc:1428] T b310367497754ac5b7132de2ef85f477 P ee93d05f71604a54a90ca3d60e798c0c: Time spent starting tablet: real 0.013s user 0.009s sys 0.003s
I20260924 10:00:38.319701 20191 raft_consensus.cc:482] T b310367497754ac5b7132de2ef85f477 P ee93d05f71604a54a90ca3d60e798c0c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 10:00:38.319947 20191 raft_consensus.cc:504] T b310367497754ac5b7132de2ef85f477 P ee93d05f71604a54a90ca3d60e798c0c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "924d4d9496ef42f0a704dec2a36af1b5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38307 } } peers { permanent_uuid: "ee93d05f71604a54a90ca3d60e798c0c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37505 } } peers { permanent_uuid: "8a8c6d9364c04579bbbf3a180ca87f9b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40969 } }
I20260924 10:00:38.321705 20191 leader_election.cc:288] T b310367497754ac5b7132de2ef85f477 P ee93d05f71604a54a90ca3d60e798c0c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 924d4d9496ef42f0a704dec2a36af1b5 (127.0.97.2:38307), 8a8c6d9364c04579bbbf3a180ca87f9b (127.0.97.1:40969)
I20260924 10:00:38.327695 20178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:38.322026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:53621 (local address 127.0.97.2:38307)
0924 10:00:38.322191 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0924 10:00:38.322195 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:00:38.322242 (+ 47us) server_negotiation.cc:407] Connection header received
0924 10:00:38.322473 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:00:38.322478 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:00:38.322548 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:00:38.322648 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:00:38.323114 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:00:38.324110 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:00:38.326004 (+ 1894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:00:38.326835 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:00:38.326953 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:00:38.327045 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0924 10:00:38.327314 (+ 269us) server_negotiation.cc:299] Negotiation successful
0924 10:00:38.327436 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 10:00:38.327692 20167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:38.321985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:38307 (local address 127.0.97.3:53621)
0924 10:00:38.322175 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0924 10:00:38.322194 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 10:00:38.322308 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:00:38.322675 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:00:38.322680 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:00:38.322707 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:00:38.322934 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:00:38.322946 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:00:38.324219 (+ 1273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:00:38.324223 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:00:38.325855 (+ 1632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:00:38.325870 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:00:38.326966 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:00:38.326971 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:00:38.327156 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:00:38.327179 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 10:00:38.327290 (+ 111us) client_negotiation.cc:240] Negotiation successful
0924 10:00:38.327423 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260924 10:00:38.328835 20182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:38.322527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:53403 (local address 127.0.97.1:40969)
0924 10:00:38.322701 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0924 10:00:38.322706 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:00:38.323301 (+ 595us) server_negotiation.cc:407] Connection header received
0924 10:00:38.323424 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:00:38.323429 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:00:38.323498 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:00:38.323589 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:00:38.324047 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:00:38.324769 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:00:38.326805 (+ 2036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:00:38.327934 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:00:38.328059 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:00:38.328132 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 10:00:38.328498 (+ 366us) server_negotiation.cc:299] Negotiation successful
0924 10:00:38.328615 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 10:00:38.328902 20192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:38.322366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40969 (local address 127.0.97.3:53403)
0924 10:00:38.323186 (+ 820us) negotiation.cc:107] Waiting for socket to connect
0924 10:00:38.323199 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 10:00:38.323335 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:00:38.323592 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:00:38.323594 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:00:38.323609 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:00:38.323905 (+ 296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:00:38.323917 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:00:38.324906 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:00:38.324911 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:00:38.326658 (+ 1747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:00:38.326673 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:00:38.328154 (+ 1481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:00:38.328160 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:00:38.328328 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:00:38.328360 (+ 32us) client_negotiation.cc:769] Sending connection context
0924 10:00:38.328465 (+ 105us) client_negotiation.cc:240] Negotiation successful
0924 10:00:38.328600 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":711,"thread_start_us":487,"threads_started":1}
I20260924 10:00:38.328760 19987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b310367497754ac5b7132de2ef85f477" candidate_uuid: "ee93d05f71604a54a90ca3d60e798c0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "924d4d9496ef42f0a704dec2a36af1b5" is_pre_election: true
I20260924 10:00:38.329252 19987 raft_consensus.cc:2454] T b310367497754ac5b7132de2ef85f477 P 924d4d9496ef42f0a704dec2a36af1b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee93d05f71604a54a90ca3d60e798c0c in term 0.
I20260924 10:00:38.329890 20054 leader_election.cc:302] T b310367497754ac5b7132de2ef85f477 P ee93d05f71604a54a90ca3d60e798c0c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 924d4d9496ef42f0a704dec2a36af1b5, ee93d05f71604a54a90ca3d60e798c0c; no voters:
I20260924 10:00:38.329988 19853 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b310367497754ac5b7132de2ef85f477" candidate_uuid: "ee93d05f71604a54a90ca3d60e798c0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a8c6d9364c04579bbbf3a180ca87f9b" is_pre_election: true
I20260924 10:00:38.330255 20191 raft_consensus.cc:2781] T b310367497754ac5b7132de2ef85f477 P ee93d05f71604a54a90ca3d60e798c0c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 10:00:38.330385 20191 raft_consensus.cc:482] T b310367497754ac5b7132de2ef85f477 P ee93d05f71604a54a90ca3d60e798c0c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 10:00:38.330473 20191 raft_consensus.cc:3037] T b310367497754ac5b7132de2ef85f477 P ee93d05f71604a54a90ca3d60e798c0c [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:00:38.330482 19853 raft_consensus.cc:2454] T b310367497754ac5b7132de2ef85f477 P 8a8c6d9364c04579bbbf3a180ca87f9b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee93d05f71604a54a90ca3d60e798c0c in term 0.
I20260924 10:00:38.332810 20191 raft_consensus.cc:504] T b310367497754ac5b7132de2ef85f477 P ee93d05f71604a54a90ca3d60e798c0c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "924d4d9496ef42f0a704dec2a36af1b5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38307 } } peers { permanent_uuid: "ee93d05f71604a54a90ca3d60e798c0c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37505 } } peers { permanent_uuid: "8a8c6d9364c04579bbbf3a180ca87f9b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40969 } }
I20260924 10:00:38.333356 20191 leader_election.cc:288] T b310367497754ac5b7132de2ef85f477 P ee93d05f71604a54a90ca3d60e798c0c [CANDIDATE]: Term 1 election: Requested vote from peers 924d4d9496ef42f0a704dec2a36af1b5 (127.0.97.2:38307), 8a8c6d9364c04579bbbf3a180ca87f9b (127.0.97.1:40969)
I20260924 10:00:38.333844 19987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b310367497754ac5b7132de2ef85f477" candidate_uuid: "ee93d05f71604a54a90ca3d60e798c0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "924d4d9496ef42f0a704dec2a36af1b5"
I20260924 10:00:38.333923 19853 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b310367497754ac5b7132de2ef85f477" candidate_uuid: "ee93d05f71604a54a90ca3d60e798c0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a8c6d9364c04579bbbf3a180ca87f9b"
I20260924 10:00:38.334105 19853 raft_consensus.cc:3037] T b310367497754ac5b7132de2ef85f477 P 8a8c6d9364c04579bbbf3a180ca87f9b [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:00:38.334095 19987 raft_consensus.cc:3037] T b310367497754ac5b7132de2ef85f477 P 924d4d9496ef42f0a704dec2a36af1b5 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:00:38.337301 19987 raft_consensus.cc:2454] T b310367497754ac5b7132de2ef85f477 P 924d4d9496ef42f0a704dec2a36af1b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee93d05f71604a54a90ca3d60e798c0c in term 1.
I20260924 10:00:38.337301 19853 raft_consensus.cc:2454] T b310367497754ac5b7132de2ef85f477 P 8a8c6d9364c04579bbbf3a180ca87f9b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee93d05f71604a54a90ca3d60e798c0c in term 1.
I20260924 10:00:38.337836 20054 leader_election.cc:302] T b310367497754ac5b7132de2ef85f477 P ee93d05f71604a54a90ca3d60e798c0c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 924d4d9496ef42f0a704dec2a36af1b5, ee93d05f71604a54a90ca3d60e798c0c; no voters:
I20260924 10:00:38.338145 20191 raft_consensus.cc:2781] T b310367497754ac5b7132de2ef85f477 P ee93d05f71604a54a90ca3d60e798c0c [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:00:38.338608 20191 raft_consensus.cc:686] T b310367497754ac5b7132de2ef85f477 P ee93d05f71604a54a90ca3d60e798c0c [term 1 LEADER]: Becoming Leader. State: Replica: ee93d05f71604a54a90ca3d60e798c0c, State: Running, Role: LEADER
I20260924 10:00:38.339166 20191 consensus_queue.cc:237] T b310367497754ac5b7132de2ef85f477 P ee93d05f71604a54a90ca3d60e798c0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "924d4d9496ef42f0a704dec2a36af1b5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38307 } } peers { permanent_uuid: "ee93d05f71604a54a90ca3d60e798c0c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37505 } } peers { permanent_uuid: "8a8c6d9364c04579bbbf3a180ca87f9b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40969 } }
I20260924 10:00:38.343678 19709 catalog_manager.cc:5495] T b310367497754ac5b7132de2ef85f477 P ee93d05f71604a54a90ca3d60e798c0c reported cstate change: term changed from 0 to 1, leader changed from <none> to ee93d05f71604a54a90ca3d60e798c0c (127.0.97.3). New cstate: current_term: 1 leader_uuid: "ee93d05f71604a54a90ca3d60e798c0c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "924d4d9496ef42f0a704dec2a36af1b5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38307 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee93d05f71604a54a90ca3d60e798c0c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37505 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a8c6d9364c04579bbbf3a180ca87f9b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40969 } health_report { overall_health: UNKNOWN } } }
I20260924 10:00:38.374331 20180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:38.369048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48148 (local address 127.0.97.3:37505)
0924 10:00:38.369247 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0924 10:00:38.369253 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:00:38.369320 (+ 67us) server_negotiation.cc:407] Connection header received
0924 10:00:38.369541 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:00:38.369545 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:00:38.369634 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:00:38.369763 (+ 129us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:00:38.370123 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:00:38.370972 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:00:38.372326 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:00:38.372720 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:00:38.372823 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:00:38.373303 (+ 480us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:00:38.373794 (+ 491us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:00:38.373934 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0924 10:00:38.374005 (+ 71us) server_negotiation.cc:299] Negotiation successful
0924 10:00:38.374083 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 10:00:38.382012 19853 raft_consensus.cc:1261] T b310367497754ac5b7132de2ef85f477 P 8a8c6d9364c04579bbbf3a180ca87f9b [term 1 FOLLOWER]: Refusing update from remote peer ee93d05f71604a54a90ca3d60e798c0c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:00:38.382009 19987 raft_consensus.cc:1261] T b310367497754ac5b7132de2ef85f477 P 924d4d9496ef42f0a704dec2a36af1b5 [term 1 FOLLOWER]: Refusing update from remote peer ee93d05f71604a54a90ca3d60e798c0c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:00:38.382889 20193 consensus_queue.cc:1035] T b310367497754ac5b7132de2ef85f477 P ee93d05f71604a54a90ca3d60e798c0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a8c6d9364c04579bbbf3a180ca87f9b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:00:38.383221 20191 consensus_queue.cc:1035] T b310367497754ac5b7132de2ef85f477 P ee93d05f71604a54a90ca3d60e798c0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "924d4d9496ef42f0a704dec2a36af1b5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:00:38.393417 20196 mvcc.cc:205] Tried to move back new op lower bound from 7332839581204144128 to 7332839581041741824. Current Snapshot: MvccSnapshot[applied={T|T < 7332839581204144128}]
I20260924 10:00:38.395145 20198 mvcc.cc:205] Tried to move back new op lower bound from 7332839581204144128 to 7332839581041741824. Current Snapshot: MvccSnapshot[applied={T|T < 7332839581204144128}]
I20260924 10:00:38.405365 19708 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54308:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:00:38.405622 19708 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54308:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:00:38.409130 19708 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 464c4899bba54f59af72bfebd7801b11: Sending DeleteTablet for 3 replicas of tablet b310367497754ac5b7132de2ef85f477
I20260924 10:00:38.410311 19967 tablet_service.cc:1514] Processing DeleteTablet for tablet b310367497754ac5b7132de2ef85f477 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:00:38 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:57240
I20260924 10:00:38.410358 20100 tablet_service.cc:1514] Processing DeleteTablet for tablet b310367497754ac5b7132de2ef85f477 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:00:38 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:48144
I20260924 10:00:38.410907 19833 tablet_service.cc:1514] Processing DeleteTablet for tablet b310367497754ac5b7132de2ef85f477 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:00:38 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:45858
I20260924 10:00:38.417929 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu pid 19770
I20260924 10:00:38.420384 20207 tablet_replica.cc:330] T b310367497754ac5b7132de2ef85f477 P 8a8c6d9364c04579bbbf3a180ca87f9b: stopping tablet replica
I20260924 10:00:38.421384 20207 raft_consensus.cc:2229] T b310367497754ac5b7132de2ef85f477 P 8a8c6d9364c04579bbbf3a180ca87f9b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:00:38.422863 20207 raft_consensus.cc:2258] T b310367497754ac5b7132de2ef85f477 P 8a8c6d9364c04579bbbf3a180ca87f9b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:00:38.426144 20207 ts_tablet_manager.cc:1905] T b310367497754ac5b7132de2ef85f477 P 8a8c6d9364c04579bbbf3a180ca87f9b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:00:38.435793 20205 tablet_replica.cc:330] T b310367497754ac5b7132de2ef85f477 P ee93d05f71604a54a90ca3d60e798c0c: stopping tablet replica
I20260924 10:00:38.436209 20206 tablet_replica.cc:330] T b310367497754ac5b7132de2ef85f477 P 924d4d9496ef42f0a704dec2a36af1b5: stopping tablet replica
I20260924 10:00:38.436262 20205 raft_consensus.cc:2229] T b310367497754ac5b7132de2ef85f477 P ee93d05f71604a54a90ca3d60e798c0c [term 1 LEADER]: Raft consensus shutting down.
I20260924 10:00:38.436744 20206 raft_consensus.cc:2229] T b310367497754ac5b7132de2ef85f477 P 924d4d9496ef42f0a704dec2a36af1b5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:00:38.436957 20205 raft_consensus.cc:2258] T b310367497754ac5b7132de2ef85f477 P ee93d05f71604a54a90ca3d60e798c0c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:00:38.437350 20206 raft_consensus.cc:2258] T b310367497754ac5b7132de2ef85f477 P 924d4d9496ef42f0a704dec2a36af1b5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:00:38.438740 20207 ts_tablet_manager.cc:1918] T b310367497754ac5b7132de2ef85f477 P 8a8c6d9364c04579bbbf3a180ca87f9b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:00:38.439261 20207 log.cc:1196] T b310367497754ac5b7132de2ef85f477 P 8a8c6d9364c04579bbbf3a180ca87f9b: Deleting WAL directory at /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790243735634835-388-0/minicluster-data/ts-0/wal/wals/b310367497754ac5b7132de2ef85f477
I20260924 10:00:38.439803 20207 ts_tablet_manager.cc:1939] T b310367497754ac5b7132de2ef85f477 P 8a8c6d9364c04579bbbf3a180ca87f9b: Deleting consensus metadata
I20260924 10:00:38.440095 20206 ts_tablet_manager.cc:1905] T b310367497754ac5b7132de2ef85f477 P 924d4d9496ef42f0a704dec2a36af1b5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:00:38.441629 19694 catalog_manager.cc:4841] TS 8a8c6d9364c04579bbbf3a180ca87f9b (127.0.97.1:40969): tablet b310367497754ac5b7132de2ef85f477 (table test-table [id=c97b3aae7e974e94a3d232c46510184f]) successfully deleted
Sep 24 10:00:38 dist-test-slave-w8d2 krb5kdc[19664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244036, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260924 10:00:38.443492 20205 ts_tablet_manager.cc:1905] T b310367497754ac5b7132de2ef85f477 P ee93d05f71604a54a90ca3d60e798c0c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:00:38.446714 20206 ts_tablet_manager.cc:1918] T b310367497754ac5b7132de2ef85f477 P 924d4d9496ef42f0a704dec2a36af1b5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:00:38.446858 20206 log.cc:1196] T b310367497754ac5b7132de2ef85f477 P 924d4d9496ef42f0a704dec2a36af1b5: Deleting WAL directory at /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790243735634835-388-0/minicluster-data/ts-1/wal/wals/b310367497754ac5b7132de2ef85f477
I20260924 10:00:38.447371 20206 ts_tablet_manager.cc:1939] T b310367497754ac5b7132de2ef85f477 P 924d4d9496ef42f0a704dec2a36af1b5: Deleting consensus metadata
I20260924 10:00:38.448987 19694 catalog_manager.cc:4841] TS 924d4d9496ef42f0a704dec2a36af1b5 (127.0.97.2:38307): tablet b310367497754ac5b7132de2ef85f477 (table test-table [id=c97b3aae7e974e94a3d232c46510184f]) successfully deleted
I20260924 10:00:38.450606 20205 ts_tablet_manager.cc:1918] T b310367497754ac5b7132de2ef85f477 P ee93d05f71604a54a90ca3d60e798c0c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:00:38.450760 20205 log.cc:1196] T b310367497754ac5b7132de2ef85f477 P ee93d05f71604a54a90ca3d60e798c0c: Deleting WAL directory at /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790243735634835-388-0/minicluster-data/ts-2/wal/wals/b310367497754ac5b7132de2ef85f477
I20260924 10:00:38.451457 20205 ts_tablet_manager.cc:1939] T b310367497754ac5b7132de2ef85f477 P ee93d05f71604a54a90ca3d60e798c0c: Deleting consensus metadata
I20260924 10:00:38.451632 20182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:38.419362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45864 (local address 127.0.97.1:40969)
0924 10:00:38.419565 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0924 10:00:38.419571 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:00:38.429651 (+ 10080us) server_negotiation.cc:407] Connection header received
0924 10:00:38.430154 (+ 503us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:00:38.430160 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:00:38.430246 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:00:38.430328 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:00:38.432280 (+ 1952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:00:38.433449 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:00:38.437290 (+ 3841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:00:38.437877 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:00:38.437967 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:00:38.444510 (+ 6543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:00:38.444526 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:00:38.444557 (+ 31us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:00:38.444600 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:00:38.448280 (+ 3680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:00:38.449202 (+ 922us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:00:38.449211 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:00:38.449214 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:00:38.449355 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:00:38.449831 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:00:38.449837 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:00:38.449841 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:00:38.450294 (+ 453us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:00:38.450583 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0924 10:00:38.451003 (+ 420us) server_negotiation.cc:299] Negotiation successful
0924 10:00:38.451216 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260924 10:00:38.452262 19893 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:00:38.453629 19695 catalog_manager.cc:4841] TS ee93d05f71604a54a90ca3d60e798c0c (127.0.97.3:37505): tablet b310367497754ac5b7132de2ef85f477 (table test-table [id=c97b3aae7e974e94a3d232c46510184f]) successfully deleted
I20260924 10:00:38.582271 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu with pid 19770
I20260924 10:00:38.596607 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu pid 19904
Sep 24 10:00:38 dist-test-slave-w8d2 krb5kdc[19664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244036, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260924 10:00:38.611521 20178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:38.597348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57250 (local address 127.0.97.2:38307)
0924 10:00:38.597535 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0924 10:00:38.597541 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:00:38.597561 (+ 20us) server_negotiation.cc:407] Connection header received
0924 10:00:38.597750 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:00:38.597756 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:00:38.597850 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:00:38.597931 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:00:38.599156 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:00:38.599943 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:00:38.601255 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:00:38.601638 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:00:38.601745 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:00:38.605975 (+ 4230us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:00:38.605989 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:00:38.606004 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:00:38.606039 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:00:38.608912 (+ 2873us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:00:38.609530 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:00:38.609538 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:00:38.609540 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:00:38.609655 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:00:38.609993 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:00:38.609997 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:00:38.609999 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:00:38.610364 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:00:38.610622 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0924 10:00:38.610921 (+ 299us) server_negotiation.cc:299] Negotiation successful
0924 10:00:38.611118 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260924 10:00:38.612136 20027 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:00:38.699482 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu with pid 19904
I20260924 10:00:38.713259 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu pid 20037
Sep 24 10:00:38 dist-test-slave-w8d2 krb5kdc[19664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244036, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260924 10:00:38.730054 20180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:38.714005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48162 (local address 127.0.97.3:37505)
0924 10:00:38.714184 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0924 10:00:38.714189 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:00:38.714217 (+ 28us) server_negotiation.cc:407] Connection header received
0924 10:00:38.714339 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:00:38.714345 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:00:38.714418 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:00:38.714500 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:00:38.715797 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:00:38.716629 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:00:38.717855 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:00:38.718301 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:00:38.718419 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:00:38.722885 (+ 4466us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:00:38.722900 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:00:38.722919 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:00:38.722966 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:00:38.726895 (+ 3929us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:00:38.727690 (+ 795us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:00:38.727700 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:00:38.727704 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:00:38.727895 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:00:38.728235 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:00:38.728240 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:00:38.728243 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:00:38.728706 (+ 463us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:00:38.729034 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0924 10:00:38.729425 (+ 391us) server_negotiation.cc:299] Negotiation successful
0924 10:00:38.729654 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 10:00:38.730826 20160 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:00:38.851331 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu with pid 20037
I20260924 10:00:38.866555 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu pid 19677
I20260924 10:00:38.867594 19739 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:00:38.975425 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu with pid 19677
2026-09-24T10:00:38Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (3964 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-24T10:00:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:00:39Z Disabled control of system clock
I20260924 10:00:39.043140 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
/tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790243735634835-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790243735634835-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790243735634835-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790243735634835-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43389
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:38249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:43389
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 10:00:39.357316 20216 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:00:39.357604 20216 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:00:39.357683 20216 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:00:39.364940 20216 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:00:39.365052 20216 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:00:39.365097 20216 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:00:39.365152 20216 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:00:39.365195 20216 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:00:39.374356 20216 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790243735634835-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790243735634835-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:43389
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43389
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790243735634835-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790243735634835-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:00:39.375627 20216 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:00:39.377321 20216 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:00:39.388362 20224 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:00:39.388435 20222 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:00:39.388391 20221 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:00:39.389292 20216 server_base.cc:1034] running on GCE node
I20260924 10:00:39.390035 20216 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:00:39.391438 20216 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:00:39.392642 20216 hybrid_clock.cc:648] HybridClock initialized: now 1790244039392587 us; error 48 us; skew 500 ppm
I20260924 10:00:39.392982 20216 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 10:00:39.395789 20216 webserver.cc:460] Webserver started at http://127.0.97.62:37251/ using document root <none> and password file <none>
I20260924 10:00:39.396457 20216 fs_manager.cc:359] Metadata directory not provided
I20260924 10:00:39.396556 20216 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:00:39.396853 20216 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:00:39.399678 20216 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790243735634835-388-0/minicluster-data/master-0/data/instance:
uuid: "0d2da31faa894407937fa13a11af9415"
format_stamp: "Formatted at 2026-09-24 10:00:39 on dist-test-slave-w8d2"
server_key: "129bf15a09270c3a08e9380605ce7937"
server_key_iv: "ff32938f08a724815ee786106bdd5dd7"
server_key_version: "encryptionkey@0"
I20260924 10:00:39.400418 20216 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790243735634835-388-0/minicluster-data/master-0/wal/instance:
uuid: "0d2da31faa894407937fa13a11af9415"
format_stamp: "Formatted at 2026-09-24 10:00:39 on dist-test-slave-w8d2"
server_key: "129bf15a09270c3a08e9380605ce7937"
server_key_iv: "ff32938f08a724815ee786106bdd5dd7"
server_key_version: "encryptionkey@0"
I20260924 10:00:39.405377 20216 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.005s
I20260924 10:00:39.408944 20230 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:00:39.410247 20216 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260924 10:00:39.410413 20216 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790243735634835-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790243735634835-388-0/minicluster-data/master-0/wal
uuid: "0d2da31faa894407937fa13a11af9415"
format_stamp: "Formatted at 2026-09-24 10:00:39 on dist-test-slave-w8d2"
server_key: "129bf15a09270c3a08e9380605ce7937"
server_key_iv: "ff32938f08a724815ee786106bdd5dd7"
server_key_version: "encryptionkey@0"
I20260924 10:00:39.410627 20216 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790243735634835-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790243735634835-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790243735634835-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:00:39.425031 20216 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:00:39.425944 20216 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:00:39.426218 20216 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:00:39.442785 20216 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:43389
I20260924 10:00:39.442806 20281 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:43389 every 8 connection(s)
I20260924 10:00:39.444104 20216 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790243735634835-388-0/minicluster-data/master-0/data/info.pb
I20260924 10:00:39.447748 20282 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:00:39.453445 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu as pid 20216
I20260924 10:00:39.453626 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790243735634835-388-0/minicluster-data/master-0/wal/instance
I20260924 10:00:39.454169 388 external_mini_cluster.cc:1428] Setting key 38b1db70230d261022c3122c2fe4531d
I20260924 10:00:39.458549 20282 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d2da31faa894407937fa13a11af9415: Bootstrap starting.
I20260924 10:00:39.461592 20282 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0d2da31faa894407937fa13a11af9415: Neither blocks nor log segments found. Creating new log.
I20260924 10:00:39.462826 20282 log.cc:824] T 00000000000000000000000000000000 P 0d2da31faa894407937fa13a11af9415: Log is configured to *not* fsync() on all Append() calls
I20260924 10:00:39.465999 20282 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d2da31faa894407937fa13a11af9415: No bootstrap required, opened a new log
I20260924 10:00:39.467921 20285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:39.456139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59432 (local address 127.0.97.62:43389)
0924 10:00:39.457112 (+ 973us) server_negotiation.cc:206] Beginning negotiation
0924 10:00:39.457123 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 10:00:39.457165 (+ 42us) server_negotiation.cc:407] Connection header received
0924 10:00:39.458866 (+ 1701us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:00:39.458889 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:00:39.459393 (+ 504us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:00:39.459770 (+ 377us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:00:39.460324 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:00:39.461600 (+ 1276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:00:39.463503 (+ 1903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:00:39.464436 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:00:39.464565 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:00:39.465647 (+ 1082us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:00:39.465655 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:00:39.465669 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:00:39.466223 (+ 554us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:00:39.466301 (+ 78us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:00:39.466323 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:00:39.466415 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0924 10:00:39.466614 (+ 199us) server_negotiation.cc:299] Negotiation successful
0924 10:00:39.466918 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":546,"thread_start_us":326,"threads_started":1}
I20260924 10:00:39.473981 20282 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0d2da31faa894407937fa13a11af9415 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d2da31faa894407937fa13a11af9415" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43389 } }
I20260924 10:00:39.474395 20282 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0d2da31faa894407937fa13a11af9415 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:00:39.474493 20282 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0d2da31faa894407937fa13a11af9415 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d2da31faa894407937fa13a11af9415, State: Initialized, Role: FOLLOWER
I20260924 10:00:39.474994 20282 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0d2da31faa894407937fa13a11af9415 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d2da31faa894407937fa13a11af9415" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43389 } }
I20260924 10:00:39.475190 20282 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0d2da31faa894407937fa13a11af9415 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:00:39.475342 20282 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0d2da31faa894407937fa13a11af9415 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:00:39.475481 20282 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0d2da31faa894407937fa13a11af9415 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:00:39.477679 20282 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0d2da31faa894407937fa13a11af9415 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d2da31faa894407937fa13a11af9415" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43389 } }
I20260924 10:00:39.478124 20282 leader_election.cc:302] T 00000000000000000000000000000000 P 0d2da31faa894407937fa13a11af9415 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0d2da31faa894407937fa13a11af9415; no voters:
I20260924 10:00:39.478641 20282 leader_election.cc:288] T 00000000000000000000000000000000 P 0d2da31faa894407937fa13a11af9415 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:00:39.478758 20287 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0d2da31faa894407937fa13a11af9415 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:00:39.480072 20287 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0d2da31faa894407937fa13a11af9415 [term 1 LEADER]: Becoming Leader. State: Replica: 0d2da31faa894407937fa13a11af9415, State: Running, Role: LEADER
I20260924 10:00:39.480654 20282 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0d2da31faa894407937fa13a11af9415 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:00:39.480633 20287 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0d2da31faa894407937fa13a11af9415 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d2da31faa894407937fa13a11af9415" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43389 } }
I20260924 10:00:39.483368 20288 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d2da31faa894407937fa13a11af9415 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0d2da31faa894407937fa13a11af9415" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d2da31faa894407937fa13a11af9415" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43389 } } }
I20260924 10:00:39.483770 20288 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d2da31faa894407937fa13a11af9415 [sys.catalog]: This master's current role is: LEADER
I20260924 10:00:39.484251 20289 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d2da31faa894407937fa13a11af9415 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0d2da31faa894407937fa13a11af9415. Latest consensus state: current_term: 1 leader_uuid: "0d2da31faa894407937fa13a11af9415" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d2da31faa894407937fa13a11af9415" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43389 } } }
I20260924 10:00:39.484551 20289 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d2da31faa894407937fa13a11af9415 [sys.catalog]: This master's current role is: LEADER
I20260924 10:00:39.485767 20296 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:00:39.490536 20296 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:00:39.498625 20296 catalog_manager.cc:1347] Generated new cluster ID: 4ab2cf5047604486bf829523f7332319
I20260924 10:00:39.498729 20296 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:00:39.522850 20296 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:00:39.524287 20296 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:00:39.533473 20296 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0d2da31faa894407937fa13a11af9415: Generated new TSK 0
I20260924 10:00:39.534278 20296 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 10:00:39.542574 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
/tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790243735634835-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790243735634835-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790243735634835-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790243735634835-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43389
--builtin_ntp_servers=127.0.97.20:38249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 10:00:39.880096 20307 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:00:39.880368 20307 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:00:39.880479 20307 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:00:39.887908 20307 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:00:39.888015 20307 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:00:39.888152 20307 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260924 10:00:39.897446 20307 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790243735634835-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790243735634835-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790243735634835-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43389
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790243735634835-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:00:39.898765 20307 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:00:39.900527 20307 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:00:39.913033 20313 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:00:39.913029 20315 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:00:39.913084 20312 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:00:39.913902 20307 server_base.cc:1034] running on GCE node
I20260924 10:00:39.914438 20307 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:00:39.915365 20307 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:00:39.916563 20307 hybrid_clock.cc:648] HybridClock initialized: now 1790244039916518 us; error 35 us; skew 500 ppm
I20260924 10:00:39.916872 20307 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 10:00:39.918936 20307 webserver.cc:460] Webserver started at http://127.0.97.1:38607/ using document root <none> and password file <none>
I20260924 10:00:39.919610 20307 fs_manager.cc:359] Metadata directory not provided
I20260924 10:00:39.919699 20307 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:00:39.919993 20307 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:00:39.922647 20307 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790243735634835-388-0/minicluster-data/ts-0/data/instance:
uuid: "fac8f88205bb434c957b2578889c0f35"
format_stamp: "Formatted at 2026-09-24 10:00:39 on dist-test-slave-w8d2"
server_key: "6e0d647ed2cbd9caf9bfaba3f482fe3d"
server_key_iv: "e2f00e0ae5e781ad2bd239be260057cf"
server_key_version: "encryptionkey@0"
I20260924 10:00:39.923421 20307 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790243735634835-388-0/minicluster-data/ts-0/wal/instance:
uuid: "fac8f88205bb434c957b2578889c0f35"
format_stamp: "Formatted at 2026-09-24 10:00:39 on dist-test-slave-w8d2"
server_key: "6e0d647ed2cbd9caf9bfaba3f482fe3d"
server_key_iv: "e2f00e0ae5e781ad2bd239be260057cf"
server_key_version: "encryptionkey@0"
I20260924 10:00:39.928328 20307 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20260924 10:00:39.931584 20321 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:00:39.932765 20307 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260924 10:00:39.932914 20307 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790243735634835-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790243735634835-388-0/minicluster-data/ts-0/wal
uuid: "fac8f88205bb434c957b2578889c0f35"
format_stamp: "Formatted at 2026-09-24 10:00:39 on dist-test-slave-w8d2"
server_key: "6e0d647ed2cbd9caf9bfaba3f482fe3d"
server_key_iv: "e2f00e0ae5e781ad2bd239be260057cf"
server_key_version: "encryptionkey@0"
I20260924 10:00:39.933102 20307 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790243735634835-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790243735634835-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790243735634835-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:00:39.947038 20307 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:00:39.947868 20307 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:00:39.948156 20307 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:00:39.949043 20307 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:00:39.950402 20307 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:00:39.950484 20307 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:00:39.950563 20307 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:00:39.950630 20307 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:00:39.978878 20307 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:42993
I20260924 10:00:39.978910 20433 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:42993 every 8 connection(s)
I20260924 10:00:39.980157 20307 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790243735634835-388-0/minicluster-data/ts-0/data/info.pb
I20260924 10:00:39.988897 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu as pid 20307
I20260924 10:00:39.989053 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790243735634835-388-0/minicluster-data/ts-0/wal/instance
I20260924 10:00:39.989444 388 external_mini_cluster.cc:1428] Setting key 44274e54f8e1f3e0d3958189dea8d417
I20260924 10:00:39.992439 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
/tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790243735634835-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790243735634835-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790243735634835-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790243735634835-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43389
--builtin_ntp_servers=127.0.97.20:38249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 10:00:39.992480 20436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:39.982625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:45123 (local address 127.0.97.62:43389)
0924 10:00:39.983013 (+ 388us) server_negotiation.cc:206] Beginning negotiation
0924 10:00:39.983018 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:00:39.984329 (+ 1311us) server_negotiation.cc:407] Connection header received
0924 10:00:39.986263 (+ 1934us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:00:39.986267 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:00:39.986333 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:00:39.986428 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:00:39.987429 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:00:39.988120 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:00:39.989670 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:00:39.990183 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:00:39.990310 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:00:39.991804 (+ 1494us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:00:39.991809 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:00:39.991812 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:00:39.991829 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:00:39.991865 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:00:39.991876 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:00:39.991944 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 10:00:39.992167 (+ 223us) server_negotiation.cc:299] Negotiation successful
0924 10:00:39.992237 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288,"thread_start_us":184,"threads_started":1}
I20260924 10:00:39.993955 20437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:39.983212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43389 (local address 127.0.97.1:45123)
0924 10:00:39.984166 (+ 954us) negotiation.cc:107] Waiting for socket to connect
0924 10:00:39.984218 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0924 10:00:39.986021 (+ 1803us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:00:39.986613 (+ 592us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:00:39.986621 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:00:39.986710 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:00:39.987293 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:00:39.987312 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:00:39.988221 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:00:39.988225 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:00:39.989539 (+ 1314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:00:39.989553 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:00:39.990387 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:00:39.990392 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:00:39.990669 (+ 277us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:00:39.991459 (+ 790us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 10:00:39.991463 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:00:39.991516 (+ 53us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 10:00:39.991614 (+ 98us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:00:39.991963 (+ 349us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:00:39.991973 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:00:39.991981 (+ 8us) client_negotiation.cc:769] Sending connection context
0924 10:00:39.992156 (+ 175us) client_negotiation.cc:240] Negotiation successful
0924 10:00:39.992551 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":241,"threads_started":1}
I20260924 10:00:39.995330 20434 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43389
I20260924 10:00:39.995628 20434 heartbeater.cc:464] Registering TS with master...
I20260924 10:00:39.996459 20434 heartbeater.cc:511] Master 127.0.97.62:43389 requested a full tablet report, sending...
I20260924 10:00:39.998425 20247 ts_manager.cc:194] Registered new tserver with Master: fac8f88205bb434c957b2578889c0f35 (127.0.97.1:42993)
I20260924 10:00:39.999908 20247 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.1:45123
W20260924 10:00:40.315315 20438 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:00:40.315625 20438 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:00:40.315745 20438 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:00:40.322917 20438 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:00:40.323030 20438 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:00:40.323174 20438 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260924 10:00:40.332537 20438 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790243735634835-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790243735634835-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790243735634835-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43389
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790243735634835-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:00:40.333846 20438 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:00:40.335695 20438 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:00:40.348666 20443 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:00:40.348629 20444 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:00:40.348688 20446 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:00:40.349676 20438 server_base.cc:1034] running on GCE node
I20260924 10:00:40.350286 20438 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:00:40.351262 20438 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:00:40.352547 20438 hybrid_clock.cc:648] HybridClock initialized: now 1790244040352443 us; error 58 us; skew 500 ppm
I20260924 10:00:40.352890 20438 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 10:00:40.355199 20438 webserver.cc:460] Webserver started at http://127.0.97.2:44421/ using document root <none> and password file <none>
I20260924 10:00:40.355922 20438 fs_manager.cc:359] Metadata directory not provided
I20260924 10:00:40.356022 20438 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:00:40.356317 20438 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:00:40.359143 20438 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790243735634835-388-0/minicluster-data/ts-1/data/instance:
uuid: "f7abcce5b01a4d14be45c3b24fc93e51"
format_stamp: "Formatted at 2026-09-24 10:00:40 on dist-test-slave-w8d2"
server_key: "df2b4e16dfe99c8c9ce0039d25cae87d"
server_key_iv: "1205231573c7c0e16ecad61642722a92"
server_key_version: "encryptionkey@0"
I20260924 10:00:40.359944 20438 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790243735634835-388-0/minicluster-data/ts-1/wal/instance:
uuid: "f7abcce5b01a4d14be45c3b24fc93e51"
format_stamp: "Formatted at 2026-09-24 10:00:40 on dist-test-slave-w8d2"
server_key: "df2b4e16dfe99c8c9ce0039d25cae87d"
server_key_iv: "1205231573c7c0e16ecad61642722a92"
server_key_version: "encryptionkey@0"
I20260924 10:00:40.364926 20438 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260924 10:00:40.368260 20452 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:00:40.369534 20438 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260924 10:00:40.369681 20438 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790243735634835-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790243735634835-388-0/minicluster-data/ts-1/wal
uuid: "f7abcce5b01a4d14be45c3b24fc93e51"
format_stamp: "Formatted at 2026-09-24 10:00:40 on dist-test-slave-w8d2"
server_key: "df2b4e16dfe99c8c9ce0039d25cae87d"
server_key_iv: "1205231573c7c0e16ecad61642722a92"
server_key_version: "encryptionkey@0"
I20260924 10:00:40.369874 20438 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790243735634835-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790243735634835-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790243735634835-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:00:40.387611 20438 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:00:40.388403 20438 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:00:40.388656 20438 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:00:40.389590 20438 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:00:40.391010 20438 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:00:40.391106 20438 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:00:40.391218 20438 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:00:40.391310 20438 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260924 10:00:40.420250 20438 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:43113
I20260924 10:00:40.420295 20564 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:43113 every 8 connection(s)
I20260924 10:00:40.421528 20438 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790243735634835-388-0/minicluster-data/ts-1/data/info.pb
I20260924 10:00:40.425657 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu as pid 20438
I20260924 10:00:40.425843 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790243735634835-388-0/minicluster-data/ts-1/wal/instance
I20260924 10:00:40.426272 388 external_mini_cluster.cc:1428] Setting key f501643cf5c3b6a6b6ca29b70fe0c257
I20260924 10:00:40.428891 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
/tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790243735634835-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790243735634835-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790243735634835-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790243735634835-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43389
--builtin_ntp_servers=127.0.97.20:38249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 10:00:40.433952 20436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:40.424196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:48389 (local address 127.0.97.62:43389)
0924 10:00:40.424368 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0924 10:00:40.424372 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:00:40.425759 (+ 1387us) server_negotiation.cc:407] Connection header received
0924 10:00:40.427785 (+ 2026us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:00:40.427790 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:00:40.427868 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:00:40.427952 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:00:40.428993 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:00:40.429716 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:00:40.431465 (+ 1749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:00:40.431851 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:00:40.431956 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:00:40.433213 (+ 1257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:00:40.433218 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:00:40.433221 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:00:40.433237 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:00:40.433269 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:00:40.433280 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:00:40.433350 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 10:00:40.433549 (+ 199us) server_negotiation.cc:299] Negotiation successful
0924 10:00:40.433645 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 10:00:40.434950 20567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:40.424719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43389 (local address 127.0.97.2:48389)
0924 10:00:40.425606 (+ 887us) negotiation.cc:107] Waiting for socket to connect
0924 10:00:40.425641 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 10:00:40.427527 (+ 1886us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:00:40.428188 (+ 661us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:00:40.428196 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:00:40.428284 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:00:40.428843 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:00:40.428859 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:00:40.429853 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:00:40.429857 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:00:40.431339 (+ 1482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:00:40.431351 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:00:40.432049 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:00:40.432056 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:00:40.432266 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:00:40.432933 (+ 667us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 10:00:40.432935 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:00:40.432972 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 10:00:40.433051 (+ 79us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:00:40.433359 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:00:40.433369 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:00:40.433377 (+ 8us) client_negotiation.cc:769] Sending connection context
0924 10:00:40.433526 (+ 149us) client_negotiation.cc:240] Negotiation successful
0924 10:00:40.433925 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":273,"threads_started":1}
I20260924 10:00:40.436367 20565 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43389
I20260924 10:00:40.436659 20565 heartbeater.cc:464] Registering TS with master...
I20260924 10:00:40.437505 20565 heartbeater.cc:511] Master 127.0.97.62:43389 requested a full tablet report, sending...
I20260924 10:00:40.438908 20247 ts_manager.cc:194] Registered new tserver with Master: f7abcce5b01a4d14be45c3b24fc93e51 (127.0.97.2:43113)
I20260924 10:00:40.439728 20247 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.2:48389
W20260924 10:00:40.741623 20568 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:00:40.741912 20568 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:00:40.742043 20568 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:00:40.751297 20568 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:00:40.751418 20568 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:00:40.751574 20568 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260924 10:00:40.760759 20568 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790243735634835-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790243735634835-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790243735634835-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43389
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790243735634835-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:00:40.762022 20568 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:00:40.763777 20568 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:00:40.776362 20574 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:00:40.776331 20573 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:00:40.776386 20576 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:00:40.777240 20568 server_base.cc:1034] running on GCE node
I20260924 10:00:40.777820 20568 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:00:40.778712 20568 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:00:40.779901 20568 hybrid_clock.cc:648] HybridClock initialized: now 1790244040779855 us; error 35 us; skew 500 ppm
I20260924 10:00:40.780179 20568 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 10:00:40.782375 20568 webserver.cc:460] Webserver started at http://127.0.97.3:33505/ using document root <none> and password file <none>
I20260924 10:00:40.783006 20568 fs_manager.cc:359] Metadata directory not provided
I20260924 10:00:40.783089 20568 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:00:40.783408 20568 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:00:40.786118 20568 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790243735634835-388-0/minicluster-data/ts-2/data/instance:
uuid: "18067cf9134e495d9f24781afa599276"
format_stamp: "Formatted at 2026-09-24 10:00:40 on dist-test-slave-w8d2"
server_key: "737c4659768f18584c25a0362d806a48"
server_key_iv: "2d82bc672bfbf7f8129c620ba07dd971"
server_key_version: "encryptionkey@0"
I20260924 10:00:40.786829 20568 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790243735634835-388-0/minicluster-data/ts-2/wal/instance:
uuid: "18067cf9134e495d9f24781afa599276"
format_stamp: "Formatted at 2026-09-24 10:00:40 on dist-test-slave-w8d2"
server_key: "737c4659768f18584c25a0362d806a48"
server_key_iv: "2d82bc672bfbf7f8129c620ba07dd971"
server_key_version: "encryptionkey@0"
I20260924 10:00:40.791641 20568 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260924 10:00:40.794790 20582 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:00:40.796026 20568 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 10:00:40.796142 20568 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790243735634835-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790243735634835-388-0/minicluster-data/ts-2/wal
uuid: "18067cf9134e495d9f24781afa599276"
format_stamp: "Formatted at 2026-09-24 10:00:40 on dist-test-slave-w8d2"
server_key: "737c4659768f18584c25a0362d806a48"
server_key_iv: "2d82bc672bfbf7f8129c620ba07dd971"
server_key_version: "encryptionkey@0"
I20260924 10:00:40.796325 20568 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790243735634835-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790243735634835-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790243735634835-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:00:40.819808 20568 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:00:40.820583 20568 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:00:40.820806 20568 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:00:40.821691 20568 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:00:40.823091 20568 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:00:40.823171 20568 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:00:40.823275 20568 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:00:40.823340 20568 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:00:40.851955 20568 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:40543
I20260924 10:00:40.852031 20694 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:40543 every 8 connection(s)
I20260924 10:00:40.853196 20568 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790243735634835-388-0/minicluster-data/ts-2/data/info.pb
I20260924 10:00:40.862452 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu as pid 20568
I20260924 10:00:40.862629 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790243735634835-388-0/minicluster-data/ts-2/wal/instance
I20260924 10:00:40.863152 388 external_mini_cluster.cc:1428] Setting key 59566c735ca53272660f8a1c07aa4062
I20260924 10:00:40.864965 20436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:40.855632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:41681 (local address 127.0.97.62:43389)
0924 10:00:40.855811 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0924 10:00:40.855815 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:00:40.857073 (+ 1258us) server_negotiation.cc:407] Connection header received
0924 10:00:40.858803 (+ 1730us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:00:40.858807 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:00:40.858874 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:00:40.858959 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:00:40.859983 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:00:40.860685 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:00:40.862206 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:00:40.862814 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:00:40.862939 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:00:40.864207 (+ 1268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:00:40.864213 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:00:40.864217 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:00:40.864247 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:00:40.864286 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:00:40.864302 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:00:40.864381 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0924 10:00:40.864574 (+ 193us) server_negotiation.cc:299] Negotiation successful
0924 10:00:40.864656 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 10:00:40.866044 20697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:40.856111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43389 (local address 127.0.97.3:41681)
0924 10:00:40.856946 (+ 835us) negotiation.cc:107] Waiting for socket to connect
0924 10:00:40.856986 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 10:00:40.858550 (+ 1564us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:00:40.859143 (+ 593us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:00:40.859152 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:00:40.859270 (+ 118us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:00:40.859833 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:00:40.859850 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:00:40.860820 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:00:40.860824 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:00:40.862063 (+ 1239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:00:40.862075 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:00:40.863007 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:00:40.863012 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:00:40.863205 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:00:40.863873 (+ 668us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 10:00:40.863876 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:00:40.863914 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 10:00:40.864013 (+ 99us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:00:40.864391 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:00:40.864399 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:00:40.864405 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 10:00:40.864537 (+ 132us) client_negotiation.cc:240] Negotiation successful
0924 10:00:40.864877 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":239,"threads_started":1}
I20260924 10:00:40.867202 20695 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43389
I20260924 10:00:40.867477 20695 heartbeater.cc:464] Registering TS with master...
I20260924 10:00:40.868129 20695 heartbeater.cc:511] Master 127.0.97.62:43389 requested a full tablet report, sending...
I20260924 10:00:40.869359 20247 ts_manager.cc:194] Registered new tserver with Master: 18067cf9134e495d9f24781afa599276 (127.0.97.3:40543)
I20260924 10:00:40.870101 20247 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.3:41681
I20260924 10:00:40.877821 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 10:00:40.887295 20436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:40.881675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59436 (local address 127.0.97.62:43389)
0924 10:00:40.881865 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0924 10:00:40.881872 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:00:40.882259 (+ 387us) server_negotiation.cc:407] Connection header received
0924 10:00:40.882405 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:00:40.882411 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:00:40.882487 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:00:40.882567 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:00:40.883013 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:00:40.883844 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:00:40.885188 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:00:40.885574 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:00:40.885677 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:00:40.886620 (+ 943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:00:40.886625 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:00:40.886628 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:00:40.886644 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:00:40.886675 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:00:40.886686 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:00:40.886759 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 10:00:40.886882 (+ 123us) server_negotiation.cc:299] Negotiation successful
0924 10:00:40.886970 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 10:00:40.890547 20247 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59436:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 10:00:40.897143 20247 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 10:00:40.920035 20707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:40.908454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42993 (local address 127.0.0.1:57796)
0924 10:00:40.909410 (+ 956us) negotiation.cc:107] Waiting for socket to connect
0924 10:00:40.909462 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0924 10:00:40.909638 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:00:40.911458 (+ 1820us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:00:40.911466 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:00:40.911500 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:00:40.911923 (+ 423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:00:40.911936 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:00:40.914290 (+ 2354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:00:40.914296 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:00:40.916882 (+ 2586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:00:40.916899 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:00:40.918989 (+ 2090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:00:40.918996 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:00:40.919267 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:00:40.919331 (+ 64us) client_negotiation.cc:769] Sending connection context
0924 10:00:40.919449 (+ 118us) client_negotiation.cc:240] Negotiation successful
0924 10:00:40.919651 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":682,"thread_start_us":520,"threads_started":1}
I20260924 10:00:40.922046 20708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:40.908703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57796 (local address 127.0.97.1:42993)
0924 10:00:40.910779 (+ 2076us) server_negotiation.cc:206] Beginning negotiation
0924 10:00:40.910788 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 10:00:40.910820 (+ 32us) server_negotiation.cc:407] Connection header received
0924 10:00:40.910923 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:00:40.910931 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:00:40.911188 (+ 257us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:00:40.912136 (+ 948us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:00:40.912354 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:00:40.914133 (+ 1779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:00:40.917189 (+ 3056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:00:40.918758 (+ 1569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:00:40.921120 (+ 2362us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:00:40.921394 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0924 10:00:40.921477 (+ 83us) server_negotiation.cc:299] Negotiation successful
0924 10:00:40.921663 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1935,"thread_start_us":1819,"threads_started":1}
I20260924 10:00:40.926296 20369 tablet_service.cc:1467] Processing CreateTablet for tablet b7579d43c3ba418b80c0d272588bb1bf (DEFAULT_TABLE table=test-table [id=7e4993df54d9402298f24d52f019cdb5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:00:40.927901 20369 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7579d43c3ba418b80c0d272588bb1bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:00:40.930305 20707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:40.910331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:43113 (local address 127.0.0.1:59456)
0924 10:00:40.920306 (+ 9975us) negotiation.cc:107] Waiting for socket to connect
0924 10:00:40.920325 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 10:00:40.920427 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:00:40.923789 (+ 3362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:00:40.923794 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:00:40.923828 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:00:40.924004 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:00:40.924016 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:00:40.925938 (+ 1922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:00:40.925943 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:00:40.927766 (+ 1823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:00:40.927780 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:00:40.929500 (+ 1720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:00:40.929505 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:00:40.929686 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:00:40.929732 (+ 46us) client_negotiation.cc:769] Sending connection context
0924 10:00:40.929818 (+ 86us) client_negotiation.cc:240] Negotiation successful
0924 10:00:40.929955 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9871,"mutex_wait_us":3,"thread_start_us":14040,"threads_started":1}
I20260924 10:00:40.930408 20712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:40.910986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59456 (local address 127.0.97.2:43113)
0924 10:00:40.923133 (+ 12147us) server_negotiation.cc:206] Beginning negotiation
0924 10:00:40.923141 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 10:00:40.923171 (+ 30us) server_negotiation.cc:407] Connection header received
0924 10:00:40.923310 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:00:40.923318 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:00:40.923575 (+ 257us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:00:40.924193 (+ 618us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:00:40.924458 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:00:40.925804 (+ 1346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:00:40.927954 (+ 2150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:00:40.929355 (+ 1401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:00:40.929463 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:00:40.929735 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0924 10:00:40.929866 (+ 131us) server_negotiation.cc:299] Negotiation successful
0924 10:00:40.930014 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11965,"thread_start_us":12496,"threads_started":1}
I20260924 10:00:40.932090 20710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:40.909904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41976 (local address 127.0.97.3:40543)
0924 10:00:40.924392 (+ 14488us) server_negotiation.cc:206] Beginning negotiation
0924 10:00:40.924399 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:00:40.924431 (+ 32us) server_negotiation.cc:407] Connection header received
0924 10:00:40.924525 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:00:40.924532 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:00:40.924768 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:00:40.924957 (+ 189us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:00:40.925500 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:00:40.926682 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:00:40.929198 (+ 2516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:00:40.931005 (+ 1807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:00:40.931137 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:00:40.931413 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0924 10:00:40.931576 (+ 163us) server_negotiation.cc:299] Negotiation successful
0924 10:00:40.931736 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14328,"thread_start_us":14155,"threads_started":1}
I20260924 10:00:40.932451 20709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:40.908673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:40543 (local address 127.0.0.1:41976)
0924 10:00:40.910229 (+ 1556us) negotiation.cc:107] Waiting for socket to connect
0924 10:00:40.910247 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 10:00:40.910352 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:00:40.924968 (+ 14616us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:00:40.924972 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:00:40.924997 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:00:40.925310 (+ 313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:00:40.925321 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:00:40.926834 (+ 1513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:00:40.926839 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:00:40.929000 (+ 2161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:00:40.929014 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:00:40.931199 (+ 2185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:00:40.931215 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:00:40.931402 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:00:40.931434 (+ 32us) client_negotiation.cc:769] Sending connection context
0924 10:00:40.931528 (+ 94us) client_negotiation.cc:240] Negotiation successful
0924 10:00:40.932155 (+ 627us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1410,"thread_start_us":1271,"threads_started":1}
I20260924 10:00:40.934026 20500 tablet_service.cc:1467] Processing CreateTablet for tablet b7579d43c3ba418b80c0d272588bb1bf (DEFAULT_TABLE table=test-table [id=7e4993df54d9402298f24d52f019cdb5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:00:40.935209 20500 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7579d43c3ba418b80c0d272588bb1bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:00:40.938275 20630 tablet_service.cc:1467] Processing CreateTablet for tablet b7579d43c3ba418b80c0d272588bb1bf (DEFAULT_TABLE table=test-table [id=7e4993df54d9402298f24d52f019cdb5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:00:40.939725 20630 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7579d43c3ba418b80c0d272588bb1bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:00:40.941282 20713 tablet_bootstrap.cc:492] T b7579d43c3ba418b80c0d272588bb1bf P fac8f88205bb434c957b2578889c0f35: Bootstrap starting.
I20260924 10:00:40.944751 20713 tablet_bootstrap.cc:654] T b7579d43c3ba418b80c0d272588bb1bf P fac8f88205bb434c957b2578889c0f35: Neither blocks nor log segments found. Creating new log.
I20260924 10:00:40.945990 20713 log.cc:824] T b7579d43c3ba418b80c0d272588bb1bf P fac8f88205bb434c957b2578889c0f35: Log is configured to *not* fsync() on all Append() calls
I20260924 10:00:40.949244 20713 tablet_bootstrap.cc:492] T b7579d43c3ba418b80c0d272588bb1bf P fac8f88205bb434c957b2578889c0f35: No bootstrap required, opened a new log
I20260924 10:00:40.949550 20713 ts_tablet_manager.cc:1397] T b7579d43c3ba418b80c0d272588bb1bf P fac8f88205bb434c957b2578889c0f35: Time spent bootstrapping tablet: real 0.008s user 0.008s sys 0.000s
I20260924 10:00:40.950111 20715 tablet_bootstrap.cc:492] T b7579d43c3ba418b80c0d272588bb1bf P f7abcce5b01a4d14be45c3b24fc93e51: Bootstrap starting.
I20260924 10:00:40.951450 20716 tablet_bootstrap.cc:492] T b7579d43c3ba418b80c0d272588bb1bf P 18067cf9134e495d9f24781afa599276: Bootstrap starting.
I20260924 10:00:40.953274 20715 tablet_bootstrap.cc:654] T b7579d43c3ba418b80c0d272588bb1bf P f7abcce5b01a4d14be45c3b24fc93e51: Neither blocks nor log segments found. Creating new log.
I20260924 10:00:40.954382 20715 log.cc:824] T b7579d43c3ba418b80c0d272588bb1bf P f7abcce5b01a4d14be45c3b24fc93e51: Log is configured to *not* fsync() on all Append() calls
I20260924 10:00:40.954790 20716 tablet_bootstrap.cc:654] T b7579d43c3ba418b80c0d272588bb1bf P 18067cf9134e495d9f24781afa599276: Neither blocks nor log segments found. Creating new log.
I20260924 10:00:40.956027 20716 log.cc:824] T b7579d43c3ba418b80c0d272588bb1bf P 18067cf9134e495d9f24781afa599276: Log is configured to *not* fsync() on all Append() calls
I20260924 10:00:40.957943 20715 tablet_bootstrap.cc:492] T b7579d43c3ba418b80c0d272588bb1bf P f7abcce5b01a4d14be45c3b24fc93e51: No bootstrap required, opened a new log
I20260924 10:00:40.958185 20715 ts_tablet_manager.cc:1397] T b7579d43c3ba418b80c0d272588bb1bf P f7abcce5b01a4d14be45c3b24fc93e51: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260924 10:00:40.958926 20716 tablet_bootstrap.cc:492] T b7579d43c3ba418b80c0d272588bb1bf P 18067cf9134e495d9f24781afa599276: No bootstrap required, opened a new log
I20260924 10:00:40.959174 20716 ts_tablet_manager.cc:1397] T b7579d43c3ba418b80c0d272588bb1bf P 18067cf9134e495d9f24781afa599276: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260924 10:00:40.959822 20713 raft_consensus.cc:348] T b7579d43c3ba418b80c0d272588bb1bf P fac8f88205bb434c957b2578889c0f35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fac8f88205bb434c957b2578889c0f35" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42993 } } peers { permanent_uuid: "18067cf9134e495d9f24781afa599276" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40543 } } peers { permanent_uuid: "f7abcce5b01a4d14be45c3b24fc93e51" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43113 } }
I20260924 10:00:40.961184 20713 raft_consensus.cc:374] T b7579d43c3ba418b80c0d272588bb1bf P fac8f88205bb434c957b2578889c0f35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:00:40.961323 20713 raft_consensus.cc:729] T b7579d43c3ba418b80c0d272588bb1bf P fac8f88205bb434c957b2578889c0f35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fac8f88205bb434c957b2578889c0f35, State: Initialized, Role: FOLLOWER
I20260924 10:00:40.961956 20713 consensus_queue.cc:260] T b7579d43c3ba418b80c0d272588bb1bf P fac8f88205bb434c957b2578889c0f35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fac8f88205bb434c957b2578889c0f35" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42993 } } peers { permanent_uuid: "18067cf9134e495d9f24781afa599276" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40543 } } peers { permanent_uuid: "f7abcce5b01a4d14be45c3b24fc93e51" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43113 } }
I20260924 10:00:40.963913 20434 heartbeater.cc:503] Master 127.0.97.62:43389 was elected leader, sending a full tablet report...
I20260924 10:00:40.964639 20713 ts_tablet_manager.cc:1428] T b7579d43c3ba418b80c0d272588bb1bf P fac8f88205bb434c957b2578889c0f35: Time spent starting tablet: real 0.015s user 0.012s sys 0.000s
I20260924 10:00:40.967286 20715 raft_consensus.cc:348] T b7579d43c3ba418b80c0d272588bb1bf P f7abcce5b01a4d14be45c3b24fc93e51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fac8f88205bb434c957b2578889c0f35" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42993 } } peers { permanent_uuid: "18067cf9134e495d9f24781afa599276" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40543 } } peers { permanent_uuid: "f7abcce5b01a4d14be45c3b24fc93e51" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43113 } }
I20260924 10:00:40.967805 20715 raft_consensus.cc:374] T b7579d43c3ba418b80c0d272588bb1bf P f7abcce5b01a4d14be45c3b24fc93e51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:00:40.967912 20715 raft_consensus.cc:729] T b7579d43c3ba418b80c0d272588bb1bf P f7abcce5b01a4d14be45c3b24fc93e51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7abcce5b01a4d14be45c3b24fc93e51, State: Initialized, Role: FOLLOWER
I20260924 10:00:40.968482 20715 consensus_queue.cc:260] T b7579d43c3ba418b80c0d272588bb1bf P f7abcce5b01a4d14be45c3b24fc93e51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fac8f88205bb434c957b2578889c0f35" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42993 } } peers { permanent_uuid: "18067cf9134e495d9f24781afa599276" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40543 } } peers { permanent_uuid: "f7abcce5b01a4d14be45c3b24fc93e51" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43113 } }
I20260924 10:00:40.968894 20716 raft_consensus.cc:348] T b7579d43c3ba418b80c0d272588bb1bf P 18067cf9134e495d9f24781afa599276 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fac8f88205bb434c957b2578889c0f35" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42993 } } peers { permanent_uuid: "18067cf9134e495d9f24781afa599276" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40543 } } peers { permanent_uuid: "f7abcce5b01a4d14be45c3b24fc93e51" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43113 } }
I20260924 10:00:40.969343 20716 raft_consensus.cc:374] T b7579d43c3ba418b80c0d272588bb1bf P 18067cf9134e495d9f24781afa599276 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:00:40.969460 20716 raft_consensus.cc:729] T b7579d43c3ba418b80c0d272588bb1bf P 18067cf9134e495d9f24781afa599276 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18067cf9134e495d9f24781afa599276, State: Initialized, Role: FOLLOWER
I20260924 10:00:40.969475 20565 heartbeater.cc:503] Master 127.0.97.62:43389 was elected leader, sending a full tablet report...
I20260924 10:00:40.970070 20716 consensus_queue.cc:260] T b7579d43c3ba418b80c0d272588bb1bf P 18067cf9134e495d9f24781afa599276 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fac8f88205bb434c957b2578889c0f35" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42993 } } peers { permanent_uuid: "18067cf9134e495d9f24781afa599276" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40543 } } peers { permanent_uuid: "f7abcce5b01a4d14be45c3b24fc93e51" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43113 } }
I20260924 10:00:40.970320 20715 ts_tablet_manager.cc:1428] T b7579d43c3ba418b80c0d272588bb1bf P f7abcce5b01a4d14be45c3b24fc93e51: Time spent starting tablet: real 0.012s user 0.010s sys 0.002s
I20260924 10:00:40.971360 20695 heartbeater.cc:503] Master 127.0.97.62:43389 was elected leader, sending a full tablet report...
I20260924 10:00:40.972069 20716 ts_tablet_manager.cc:1428] T b7579d43c3ba418b80c0d272588bb1bf P 18067cf9134e495d9f24781afa599276: Time spent starting tablet: real 0.013s user 0.005s sys 0.007s
W20260924 10:00:40.982205 20435 tablet.cc:2369] T b7579d43c3ba418b80c0d272588bb1bf P fac8f88205bb434c957b2578889c0f35: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 10:00:41.098269 20720 raft_consensus.cc:482] T b7579d43c3ba418b80c0d272588bb1bf P f7abcce5b01a4d14be45c3b24fc93e51 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 10:00:41.098569 20720 raft_consensus.cc:504] T b7579d43c3ba418b80c0d272588bb1bf P f7abcce5b01a4d14be45c3b24fc93e51 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fac8f88205bb434c957b2578889c0f35" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42993 } } peers { permanent_uuid: "18067cf9134e495d9f24781afa599276" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40543 } } peers { permanent_uuid: "f7abcce5b01a4d14be45c3b24fc93e51" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43113 } }
I20260924 10:00:41.100327 20720 leader_election.cc:288] T b7579d43c3ba418b80c0d272588bb1bf P f7abcce5b01a4d14be45c3b24fc93e51 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fac8f88205bb434c957b2578889c0f35 (127.0.97.1:42993), 18067cf9134e495d9f24781afa599276 (127.0.97.3:40543)
W20260924 10:00:41.104736 20696 tablet.cc:2369] T b7579d43c3ba418b80c0d272588bb1bf P 18067cf9134e495d9f24781afa599276: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 10:00:41.107180 20710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:41.100771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:40279 (local address 127.0.97.3:40543)
0924 10:00:41.100966 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0924 10:00:41.100972 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:00:41.101656 (+ 684us) server_negotiation.cc:407] Connection header received
0924 10:00:41.101872 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:00:41.101877 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:00:41.101954 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:00:41.102037 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:00:41.102780 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:00:41.103521 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:00:41.105520 (+ 1999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:00:41.106365 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:00:41.106473 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:00:41.106581 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0924 10:00:41.106823 (+ 242us) server_negotiation.cc:299] Negotiation successful
0924 10:00:41.106945 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 10:00:41.107316 20722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:41.100770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:40543 (local address 127.0.97.2:40279)
0924 10:00:41.101508 (+ 738us) negotiation.cc:107] Waiting for socket to connect
0924 10:00:41.101549 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 10:00:41.101723 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:00:41.102111 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:00:41.102115 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:00:41.102139 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:00:41.102622 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:00:41.102635 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:00:41.103637 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:00:41.103642 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:00:41.105329 (+ 1687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:00:41.105342 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:00:41.106534 (+ 1192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:00:41.106540 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:00:41.106685 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:00:41.106709 (+ 24us) client_negotiation.cc:769] Sending connection context
0924 10:00:41.106786 (+ 77us) client_negotiation.cc:240] Negotiation successful
0924 10:00:41.106924 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":583,"mutex_wait_us":105,"spinlock_wait_cycles":37504,"thread_start_us":365,"threads_started":1}
I20260924 10:00:41.107580 20708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:41.100842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:47483 (local address 127.0.97.1:42993)
0924 10:00:41.101019 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0924 10:00:41.101025 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:00:41.101909 (+ 884us) server_negotiation.cc:407] Connection header received
0924 10:00:41.102062 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:00:41.102067 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:00:41.102152 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:00:41.102245 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:00:41.102855 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:00:41.103577 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:00:41.105726 (+ 2149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:00:41.106688 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:00:41.106802 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:00:41.106885 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0924 10:00:41.107199 (+ 314us) server_negotiation.cc:299] Negotiation successful
0924 10:00:41.107347 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 10:00:41.108099 20723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:41.101118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42993 (local address 127.0.97.2:47483)
0924 10:00:41.101803 (+ 685us) negotiation.cc:107] Waiting for socket to connect
0924 10:00:41.101820 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 10:00:41.101938 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:00:41.102270 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:00:41.102272 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:00:41.102303 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:00:41.102679 (+ 376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:00:41.102687 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:00:41.103700 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:00:41.103704 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:00:41.105563 (+ 1859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:00:41.105574 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:00:41.106865 (+ 1291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:00:41.106872 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:00:41.107024 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:00:41.107052 (+ 28us) client_negotiation.cc:769] Sending connection context
0924 10:00:41.107749 (+ 697us) client_negotiation.cc:240] Negotiation successful
0924 10:00:41.107862 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":578,"mutex_wait_us":74,"thread_start_us":391,"threads_started":1}
I20260924 10:00:41.108467 20650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7579d43c3ba418b80c0d272588bb1bf" candidate_uuid: "f7abcce5b01a4d14be45c3b24fc93e51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18067cf9134e495d9f24781afa599276" is_pre_election: true
I20260924 10:00:41.108919 20650 raft_consensus.cc:2454] T b7579d43c3ba418b80c0d272588bb1bf P 18067cf9134e495d9f24781afa599276 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7abcce5b01a4d14be45c3b24fc93e51 in term 0.
I20260924 10:00:41.108952 20389 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7579d43c3ba418b80c0d272588bb1bf" candidate_uuid: "f7abcce5b01a4d14be45c3b24fc93e51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fac8f88205bb434c957b2578889c0f35" is_pre_election: true
I20260924 10:00:41.109398 20389 raft_consensus.cc:2454] T b7579d43c3ba418b80c0d272588bb1bf P fac8f88205bb434c957b2578889c0f35 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7abcce5b01a4d14be45c3b24fc93e51 in term 0.
I20260924 10:00:41.109557 20455 leader_election.cc:302] T b7579d43c3ba418b80c0d272588bb1bf P f7abcce5b01a4d14be45c3b24fc93e51 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18067cf9134e495d9f24781afa599276, f7abcce5b01a4d14be45c3b24fc93e51; no voters:
I20260924 10:00:41.109938 20720 raft_consensus.cc:2781] T b7579d43c3ba418b80c0d272588bb1bf P f7abcce5b01a4d14be45c3b24fc93e51 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 10:00:41.110074 20720 raft_consensus.cc:482] T b7579d43c3ba418b80c0d272588bb1bf P f7abcce5b01a4d14be45c3b24fc93e51 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 10:00:41.110163 20720 raft_consensus.cc:3037] T b7579d43c3ba418b80c0d272588bb1bf P f7abcce5b01a4d14be45c3b24fc93e51 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:00:41.112339 20720 raft_consensus.cc:504] T b7579d43c3ba418b80c0d272588bb1bf P f7abcce5b01a4d14be45c3b24fc93e51 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fac8f88205bb434c957b2578889c0f35" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42993 } } peers { permanent_uuid: "18067cf9134e495d9f24781afa599276" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40543 } } peers { permanent_uuid: "f7abcce5b01a4d14be45c3b24fc93e51" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43113 } }
I20260924 10:00:41.112929 20720 leader_election.cc:288] T b7579d43c3ba418b80c0d272588bb1bf P f7abcce5b01a4d14be45c3b24fc93e51 [CANDIDATE]: Term 1 election: Requested vote from peers fac8f88205bb434c957b2578889c0f35 (127.0.97.1:42993), 18067cf9134e495d9f24781afa599276 (127.0.97.3:40543)
I20260924 10:00:41.113359 20389 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7579d43c3ba418b80c0d272588bb1bf" candidate_uuid: "f7abcce5b01a4d14be45c3b24fc93e51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fac8f88205bb434c957b2578889c0f35"
I20260924 10:00:41.113547 20389 raft_consensus.cc:3037] T b7579d43c3ba418b80c0d272588bb1bf P fac8f88205bb434c957b2578889c0f35 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:00:41.113585 20650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7579d43c3ba418b80c0d272588bb1bf" candidate_uuid: "f7abcce5b01a4d14be45c3b24fc93e51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18067cf9134e495d9f24781afa599276"
I20260924 10:00:41.113778 20650 raft_consensus.cc:3037] T b7579d43c3ba418b80c0d272588bb1bf P 18067cf9134e495d9f24781afa599276 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:00:41.115800 20389 raft_consensus.cc:2454] T b7579d43c3ba418b80c0d272588bb1bf P fac8f88205bb434c957b2578889c0f35 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7abcce5b01a4d14be45c3b24fc93e51 in term 1.
I20260924 10:00:41.116026 20650 raft_consensus.cc:2454] T b7579d43c3ba418b80c0d272588bb1bf P 18067cf9134e495d9f24781afa599276 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7abcce5b01a4d14be45c3b24fc93e51 in term 1.
I20260924 10:00:41.116132 20454 leader_election.cc:302] T b7579d43c3ba418b80c0d272588bb1bf P f7abcce5b01a4d14be45c3b24fc93e51 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f7abcce5b01a4d14be45c3b24fc93e51, fac8f88205bb434c957b2578889c0f35; no voters:
I20260924 10:00:41.116406 20720 raft_consensus.cc:2781] T b7579d43c3ba418b80c0d272588bb1bf P f7abcce5b01a4d14be45c3b24fc93e51 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:00:41.116787 20720 raft_consensus.cc:686] T b7579d43c3ba418b80c0d272588bb1bf P f7abcce5b01a4d14be45c3b24fc93e51 [term 1 LEADER]: Becoming Leader. State: Replica: f7abcce5b01a4d14be45c3b24fc93e51, State: Running, Role: LEADER
I20260924 10:00:41.117229 20720 consensus_queue.cc:237] T b7579d43c3ba418b80c0d272588bb1bf P f7abcce5b01a4d14be45c3b24fc93e51 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fac8f88205bb434c957b2578889c0f35" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42993 } } peers { permanent_uuid: "18067cf9134e495d9f24781afa599276" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40543 } } peers { permanent_uuid: "f7abcce5b01a4d14be45c3b24fc93e51" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43113 } }
I20260924 10:00:41.121549 20247 catalog_manager.cc:5495] T b7579d43c3ba418b80c0d272588bb1bf P f7abcce5b01a4d14be45c3b24fc93e51 reported cstate change: term changed from 0 to 1, leader changed from <none> to f7abcce5b01a4d14be45c3b24fc93e51 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "f7abcce5b01a4d14be45c3b24fc93e51" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fac8f88205bb434c957b2578889c0f35" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42993 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "18067cf9134e495d9f24781afa599276" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40543 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f7abcce5b01a4d14be45c3b24fc93e51" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43113 } health_report { overall_health: HEALTHY } } }
I20260924 10:00:41.164283 20712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:41.159195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59460 (local address 127.0.97.2:43113)
0924 10:00:41.159385 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0924 10:00:41.159390 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:00:41.159468 (+ 78us) server_negotiation.cc:407] Connection header received
0924 10:00:41.159675 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:00:41.159679 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:00:41.159768 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:00:41.159890 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:00:41.160234 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:00:41.160916 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:00:41.162261 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:00:41.162664 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:00:41.162770 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:00:41.163218 (+ 448us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:00:41.163761 (+ 543us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:00:41.163889 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0924 10:00:41.163967 (+ 78us) server_negotiation.cc:299] Negotiation successful
0924 10:00:41.164041 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260924 10:00:41.171944 20389 raft_consensus.cc:1261] T b7579d43c3ba418b80c0d272588bb1bf P fac8f88205bb434c957b2578889c0f35 [term 1 FOLLOWER]: Refusing update from remote peer f7abcce5b01a4d14be45c3b24fc93e51: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:00:41.171944 20650 raft_consensus.cc:1261] T b7579d43c3ba418b80c0d272588bb1bf P 18067cf9134e495d9f24781afa599276 [term 1 FOLLOWER]: Refusing update from remote peer f7abcce5b01a4d14be45c3b24fc93e51: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:00:41.172835 20720 consensus_queue.cc:1035] T b7579d43c3ba418b80c0d272588bb1bf P f7abcce5b01a4d14be45c3b24fc93e51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "18067cf9134e495d9f24781afa599276" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40543 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:00:41.173097 20724 consensus_queue.cc:1035] T b7579d43c3ba418b80c0d272588bb1bf P f7abcce5b01a4d14be45c3b24fc93e51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fac8f88205bb434c957b2578889c0f35" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 10:00:41.173725 20566 tablet.cc:2369] T b7579d43c3ba418b80c0d272588bb1bf P f7abcce5b01a4d14be45c3b24fc93e51: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 10:00:41.185663 20728 mvcc.cc:205] Tried to move back new op lower bound from 7332839592631627776 to 7332839592420196352. Current Snapshot: MvccSnapshot[applied={T|T < 7332839592631627776}]
I20260924 10:00:41.196417 20246 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:59436:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:00:41.196720 20246 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:59436:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:00:41.200425 20246 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0d2da31faa894407937fa13a11af9415: Sending DeleteTablet for 3 replicas of tablet b7579d43c3ba418b80c0d272588bb1bf
I20260924 10:00:41.201613 20369 tablet_service.cc:1514] Processing DeleteTablet for tablet b7579d43c3ba418b80c0d272588bb1bf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:00:41 UTC) from {username='slave'} at 127.0.0.1:57796
I20260924 10:00:41.201753 20500 tablet_service.cc:1514] Processing DeleteTablet for tablet b7579d43c3ba418b80c0d272588bb1bf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:00:41 UTC) from {username='slave'} at 127.0.0.1:59456
I20260924 10:00:41.201835 20630 tablet_service.cc:1514] Processing DeleteTablet for tablet b7579d43c3ba418b80c0d272588bb1bf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:00:41 UTC) from {username='slave'} at 127.0.0.1:41976
I20260924 10:00:41.203722 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu pid 20307
I20260924 10:00:41.214253 20736 tablet_replica.cc:330] T b7579d43c3ba418b80c0d272588bb1bf P fac8f88205bb434c957b2578889c0f35: stopping tablet replica
I20260924 10:00:41.214769 20736 raft_consensus.cc:2229] T b7579d43c3ba418b80c0d272588bb1bf P fac8f88205bb434c957b2578889c0f35 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:00:41.215366 20736 raft_consensus.cc:2258] T b7579d43c3ba418b80c0d272588bb1bf P fac8f88205bb434c957b2578889c0f35 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:00:41.217000 20738 tablet_replica.cc:330] T b7579d43c3ba418b80c0d272588bb1bf P 18067cf9134e495d9f24781afa599276: stopping tablet replica
I20260924 10:00:41.217401 20738 raft_consensus.cc:2229] T b7579d43c3ba418b80c0d272588bb1bf P 18067cf9134e495d9f24781afa599276 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:00:41.217895 20738 raft_consensus.cc:2258] T b7579d43c3ba418b80c0d272588bb1bf P 18067cf9134e495d9f24781afa599276 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:00:41.218706 20737 tablet_replica.cc:330] T b7579d43c3ba418b80c0d272588bb1bf P f7abcce5b01a4d14be45c3b24fc93e51: stopping tablet replica
I20260924 10:00:41.219040 20737 raft_consensus.cc:2229] T b7579d43c3ba418b80c0d272588bb1bf P f7abcce5b01a4d14be45c3b24fc93e51 [term 1 LEADER]: Raft consensus shutting down.
I20260924 10:00:41.219306 20736 ts_tablet_manager.cc:1905] T b7579d43c3ba418b80c0d272588bb1bf P fac8f88205bb434c957b2578889c0f35: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:00:41.220604 20737 raft_consensus.cc:2258] T b7579d43c3ba418b80c0d272588bb1bf P f7abcce5b01a4d14be45c3b24fc93e51 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:00:41.221174 20738 ts_tablet_manager.cc:1905] T b7579d43c3ba418b80c0d272588bb1bf P 18067cf9134e495d9f24781afa599276: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:00:41.225090 20737 ts_tablet_manager.cc:1905] T b7579d43c3ba418b80c0d272588bb1bf P f7abcce5b01a4d14be45c3b24fc93e51: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:00:41.227437 20708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:41.205125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57800 (local address 127.0.97.1:42993)
0924 10:00:41.205317 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0924 10:00:41.205325 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 10:00:41.207773 (+ 2448us) server_negotiation.cc:407] Connection header received
0924 10:00:41.219584 (+ 11811us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:00:41.219590 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:00:41.219693 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:00:41.219815 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:00:41.220293 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:00:41.221378 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:00:41.223322 (+ 1944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:00:41.223933 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:00:41.224042 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:00:41.225398 (+ 1356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:00:41.225408 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:00:41.225426 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:00:41.226228 (+ 802us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:00:41.226290 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:00:41.226316 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:00:41.226429 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0924 10:00:41.226925 (+ 496us) server_negotiation.cc:299] Negotiation successful
0924 10:00:41.227031 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 10:00:41.228232 20429 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:00:41.228241 20738 ts_tablet_manager.cc:1918] T b7579d43c3ba418b80c0d272588bb1bf P 18067cf9134e495d9f24781afa599276: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:00:41.228380 20738 log.cc:1196] T b7579d43c3ba418b80c0d272588bb1bf P 18067cf9134e495d9f24781afa599276: Deleting WAL directory at /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790243735634835-388-0/minicluster-data/ts-2/wal/wals/b7579d43c3ba418b80c0d272588bb1bf
I20260924 10:00:41.228894 20738 ts_tablet_manager.cc:1939] T b7579d43c3ba418b80c0d272588bb1bf P 18067cf9134e495d9f24781afa599276: Deleting consensus metadata
I20260924 10:00:41.231025 20233 catalog_manager.cc:4841] TS 18067cf9134e495d9f24781afa599276 (127.0.97.3:40543): tablet b7579d43c3ba418b80c0d272588bb1bf (table test-table [id=7e4993df54d9402298f24d52f019cdb5]) successfully deleted
I20260924 10:00:41.235937 20737 ts_tablet_manager.cc:1918] T b7579d43c3ba418b80c0d272588bb1bf P f7abcce5b01a4d14be45c3b24fc93e51: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:00:41.236128 20737 log.cc:1196] T b7579d43c3ba418b80c0d272588bb1bf P f7abcce5b01a4d14be45c3b24fc93e51: Deleting WAL directory at /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790243735634835-388-0/minicluster-data/ts-1/wal/wals/b7579d43c3ba418b80c0d272588bb1bf
I20260924 10:00:41.236693 20737 ts_tablet_manager.cc:1939] T b7579d43c3ba418b80c0d272588bb1bf P f7abcce5b01a4d14be45c3b24fc93e51: Deleting consensus metadata
I20260924 10:00:41.238520 20233 catalog_manager.cc:4841] TS f7abcce5b01a4d14be45c3b24fc93e51 (127.0.97.2:43113): tablet b7579d43c3ba418b80c0d272588bb1bf (table test-table [id=7e4993df54d9402298f24d52f019cdb5]) successfully deleted
I20260924 10:00:41.319197 20736 ts_tablet_manager.cc:1918] T b7579d43c3ba418b80c0d272588bb1bf P fac8f88205bb434c957b2578889c0f35: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:00:41.319542 20736 log.cc:1196] T b7579d43c3ba418b80c0d272588bb1bf P fac8f88205bb434c957b2578889c0f35: Deleting WAL directory at /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790243735634835-388-0/minicluster-data/ts-0/wal/wals/b7579d43c3ba418b80c0d272588bb1bf
I20260924 10:00:41.319979 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu with pid 20307
W20260924 10:00:41.334389 20232 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:42993 (error 108)
I20260924 10:00:41.334590 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu pid 20438
I20260924 10:00:41.337360 20709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:41.336845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42993 (local address 127.0.0.1:57812)
0924 10:00:41.337051 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0924 10:00:41.337186 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260924 10:00:41.337657 20232 catalog_manager.cc:4584] TS fac8f88205bb434c957b2578889c0f35 (127.0.97.1:42993): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b7579d43c3ba418b80c0d272588bb1bf: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42993: connect: Connection refused (error 111)
I20260924 10:00:41.341777 20712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:41.335407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59472 (local address 127.0.97.2:43113)
0924 10:00:41.335618 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0924 10:00:41.335624 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:00:41.335921 (+ 297us) server_negotiation.cc:407] Connection header received
0924 10:00:41.335996 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:00:41.336001 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:00:41.336115 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:00:41.336212 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:00:41.336653 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:00:41.337563 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:00:41.338832 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:00:41.339274 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:00:41.339392 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:00:41.340404 (+ 1012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:00:41.340413 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:00:41.340427 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:00:41.341046 (+ 619us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:00:41.341096 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:00:41.341115 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:00:41.341203 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0924 10:00:41.341342 (+ 139us) server_negotiation.cc:299] Negotiation successful
0924 10:00:41.341451 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260924 10:00:41.342504 20560 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:00:41.391161 20709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:41.390713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42993 (local address 127.0.0.1:57818)
0924 10:00:41.390908 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0924 10:00:41.391000 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260924 10:00:41.392155 20709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:41.391819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42993 (local address 127.0.0.1:57822)
0924 10:00:41.391971 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0924 10:00:41.392043 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260924 10:00:41.452844 20709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:41.452375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42993 (local address 127.0.0.1:57826)
0924 10:00:41.452574 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0924 10:00:41.452664 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260924 10:00:41.454016 20709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:41.453669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42993 (local address 127.0.0.1:57838)
0924 10:00:41.453802 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0924 10:00:41.453873 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260924 10:00:41.454648 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu with pid 20438
I20260924 10:00:41.472122 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu pid 20568
I20260924 10:00:41.478258 20710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:41.472856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41980 (local address 127.0.97.3:40543)
0924 10:00:41.473055 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0924 10:00:41.473059 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:00:41.473077 (+ 18us) server_negotiation.cc:407] Connection header received
0924 10:00:41.473215 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:00:41.473219 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:00:41.473311 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:00:41.473391 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:00:41.473738 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:00:41.474460 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:00:41.475710 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:00:41.476121 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:00:41.476212 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:00:41.477119 (+ 907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:00:41.477128 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:00:41.477143 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:00:41.477685 (+ 542us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:00:41.477731 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:00:41.477749 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:00:41.477838 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0924 10:00:41.477902 (+ 64us) server_negotiation.cc:299] Negotiation successful
0924 10:00:41.478011 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 10:00:41.478853 20690 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:00:41.535192 20709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:41.534709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42993 (local address 127.0.0.1:57848)
0924 10:00:41.534920 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0924 10:00:41.535008 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260924 10:00:41.536209 20709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:41.535824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42993 (local address 127.0.0.1:57858)
0924 10:00:41.536033 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0924 10:00:41.536102 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
I20260924 10:00:41.565496 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu with pid 20568
I20260924 10:00:41.578665 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu pid 20216
I20260924 10:00:41.579712 20277 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:00:41.686228 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu with pid 20216
2026-09-24T10:00:41Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (2709 ms)
[----------] 6 tests from EncryptionPolicyTest (20539 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790243735634835-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:00:41 dist-test-slave-w8d2 krb5kdc[20747](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:00:41 dist-test-slave-w8d2 krb5kdc[20747](info): set up 2 sockets
Sep 24 10:00:41 dist-test-slave-w8d2 krb5kdc[20747](info): commencing operation
krb5kdc: starting...
W20260924 10:00:42.703512 388 mini_kdc.cc:121] Time spent starting KDC: real 0.980s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790243735634835-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790243735634835-388-0/krb5kdc/test-user.keytab.
Sep 24 10:00:42 dist-test-slave-w8d2 krb5kdc[20747](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244042, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:00:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:00:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790243735634835-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790243735634835-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 10:00:43.002890 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
/tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790243735634835-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790243735634835-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790243735634835-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790243735634835-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44759
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790243735634835-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44759
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790243735634835-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790243735634835-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790243735634835-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:00:43.317970 20760 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:00:43.318257 20760 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:00:43.318354 20760 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:00:43.325863 20760 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:00:43.325984 20760 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:00:43.326045 20760 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:00:43.326103 20760 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:00:43.326145 20760 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:00:43.336071 20760 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790243735634835-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790243735634835-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44759
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790243735634835-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44759
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790243735634835-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790243735634835-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:00:43.337322 20760 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:00:43.338562 20760 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:00:43.349589 20765 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:00:43.349563 20766 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:00:43.349613 20768 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:00:43.350530 20760 server_base.cc:1034] running on GCE node
I20260924 10:00:43.351306 20760 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:00:43.352738 20760 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:00:43.353929 20760 hybrid_clock.cc:648] HybridClock initialized: now 1790244043353875 us; error 52 us; skew 500 ppm
I20260924 10:00:43.354251 20760 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:00:43 dist-test-slave-w8d2 krb5kdc[20747](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244043, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:00:43.357245 20760 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 10:00:43.358618 20760 webserver.cc:460] Webserver started at http://127.0.0.1:33483/ using document root <none> and password file <none>
I20260924 10:00:43.359300 20760 fs_manager.cc:359] Metadata directory not provided
I20260924 10:00:43.359418 20760 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:00:43.359699 20760 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:00:43.362494 20760 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790243735634835-388-0/minicluster-data/master-0/data/instance:
uuid: "dd64f7a3058f46d8a3a42a96fb63798a"
format_stamp: "Formatted at 2026-09-24 10:00:43 on dist-test-slave-w8d2"
server_key: "c2dd008c891a1ea82d283df8dd416bce"
server_key_iv: "f3547ed78fd2318136e9450b258fa0e6"
server_key_version: "encryptionkey@0"
I20260924 10:00:43.363205 20760 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790243735634835-388-0/minicluster-data/master-0/wal/instance:
uuid: "dd64f7a3058f46d8a3a42a96fb63798a"
format_stamp: "Formatted at 2026-09-24 10:00:43 on dist-test-slave-w8d2"
server_key: "c2dd008c891a1ea82d283df8dd416bce"
server_key_iv: "f3547ed78fd2318136e9450b258fa0e6"
server_key_version: "encryptionkey@0"
I20260924 10:00:43.368261 20760 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260924 10:00:43.371496 20775 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:00:43.372785 20760 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260924 10:00:43.372937 20760 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790243735634835-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790243735634835-388-0/minicluster-data/master-0/wal
uuid: "dd64f7a3058f46d8a3a42a96fb63798a"
format_stamp: "Formatted at 2026-09-24 10:00:43 on dist-test-slave-w8d2"
server_key: "c2dd008c891a1ea82d283df8dd416bce"
server_key_iv: "f3547ed78fd2318136e9450b258fa0e6"
server_key_version: "encryptionkey@0"
I20260924 10:00:43.373111 20760 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790243735634835-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790243735634835-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790243735634835-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:00:43.397245 20760 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:00:43.401507 20760 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:00:43.401767 20760 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:00:43.418442 20760 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44759
I20260924 10:00:43.418473 20826 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44759 every 8 connection(s)
I20260924 10:00:43.419781 20760 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790243735634835-388-0/minicluster-data/master-0/data/info.pb
I20260924 10:00:43.423434 20827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:00:43.424674 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu as pid 20760
I20260924 10:00:43.424852 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790243735634835-388-0/minicluster-data/master-0/wal/instance
I20260924 10:00:43.425433 388 external_mini_cluster.cc:1428] Setting key e8f72aa6a3303482070217d2f76b41e4
I20260924 10:00:43.437345 20827 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd64f7a3058f46d8a3a42a96fb63798a: Bootstrap starting.
Sep 24 10:00:43 dist-test-slave-w8d2 krb5kdc[20747](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244042, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 10:00:43.440346 20827 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dd64f7a3058f46d8a3a42a96fb63798a: Neither blocks nor log segments found. Creating new log.
I20260924 10:00:43.441612 20827 log.cc:824] T 00000000000000000000000000000000 P dd64f7a3058f46d8a3a42a96fb63798a: Log is configured to *not* fsync() on all Append() calls
I20260924 10:00:43.444824 20827 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd64f7a3058f46d8a3a42a96fb63798a: No bootstrap required, opened a new log
I20260924 10:00:43.448403 20830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:43.427358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45019 (local address 127.0.0.1:44759)
0924 10:00:43.428140 (+ 782us) server_negotiation.cc:206] Beginning negotiation
0924 10:00:43.428159 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0924 10:00:43.428209 (+ 50us) server_negotiation.cc:407] Connection header received
0924 10:00:43.429896 (+ 1687us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:00:43.429929 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:00:43.430484 (+ 555us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:00:43.430950 (+ 466us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:00:43.432333 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:00:43.434285 (+ 1952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:00:43.435565 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:00:43.436549 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:00:43.436716 (+ 167us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:00:43.440362 (+ 3646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:00:43.440374 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:00:43.440392 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:00:43.440436 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:00:43.443872 (+ 3436us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:00:43.444616 (+ 744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:00:43.444624 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:00:43.444631 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:00:43.444835 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:00:43.445254 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:00:43.445259 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:00:43.445262 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:00:43.445921 (+ 659us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:00:43.446227 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0924 10:00:43.446634 (+ 407us) server_negotiation.cc:299] Negotiation successful
0924 10:00:43.447015 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":552,"thread_start_us":341,"threads_started":1}
I20260924 10:00:43.452113 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 10:00:43.453693 20827 raft_consensus.cc:348] T 00000000000000000000000000000000 P dd64f7a3058f46d8a3a42a96fb63798a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd64f7a3058f46d8a3a42a96fb63798a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44759 } }
I20260924 10:00:43.454028 20827 raft_consensus.cc:374] T 00000000000000000000000000000000 P dd64f7a3058f46d8a3a42a96fb63798a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:00:43.454124 20827 raft_consensus.cc:729] T 00000000000000000000000000000000 P dd64f7a3058f46d8a3a42a96fb63798a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd64f7a3058f46d8a3a42a96fb63798a, State: Initialized, Role: FOLLOWER
I20260924 10:00:43.454774 20827 consensus_queue.cc:260] T 00000000000000000000000000000000 P dd64f7a3058f46d8a3a42a96fb63798a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd64f7a3058f46d8a3a42a96fb63798a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44759 } }
I20260924 10:00:43.455021 20827 raft_consensus.cc:388] T 00000000000000000000000000000000 P dd64f7a3058f46d8a3a42a96fb63798a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:00:43.455178 20827 raft_consensus.cc:482] T 00000000000000000000000000000000 P dd64f7a3058f46d8a3a42a96fb63798a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:00:43.455379 20827 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dd64f7a3058f46d8a3a42a96fb63798a [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:00:43.457993 20827 raft_consensus.cc:504] T 00000000000000000000000000000000 P dd64f7a3058f46d8a3a42a96fb63798a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd64f7a3058f46d8a3a42a96fb63798a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44759 } }
I20260924 10:00:43.458475 20827 leader_election.cc:302] T 00000000000000000000000000000000 P dd64f7a3058f46d8a3a42a96fb63798a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dd64f7a3058f46d8a3a42a96fb63798a; no voters:
I20260924 10:00:43.459084 20827 leader_election.cc:288] T 00000000000000000000000000000000 P dd64f7a3058f46d8a3a42a96fb63798a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:00:43.459180 20837 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dd64f7a3058f46d8a3a42a96fb63798a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:00:43.461309 20837 raft_consensus.cc:686] T 00000000000000000000000000000000 P dd64f7a3058f46d8a3a42a96fb63798a [term 1 LEADER]: Becoming Leader. State: Replica: dd64f7a3058f46d8a3a42a96fb63798a, State: Running, Role: LEADER
I20260924 10:00:43.461396 20827 sys_catalog.cc:564] T 00000000000000000000000000000000 P dd64f7a3058f46d8a3a42a96fb63798a [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:00:43.461987 20837 consensus_queue.cc:237] T 00000000000000000000000000000000 P dd64f7a3058f46d8a3a42a96fb63798a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd64f7a3058f46d8a3a42a96fb63798a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44759 } }
I20260924 10:00:43.464885 20838 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd64f7a3058f46d8a3a42a96fb63798a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dd64f7a3058f46d8a3a42a96fb63798a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd64f7a3058f46d8a3a42a96fb63798a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44759 } } }
I20260924 10:00:43.465317 20838 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd64f7a3058f46d8a3a42a96fb63798a [sys.catalog]: This master's current role is: LEADER
I20260924 10:00:43.472628 20830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:43.456642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55929 (local address 127.0.0.1:44759)
0924 10:00:43.456817 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0924 10:00:43.456823 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:00:43.457074 (+ 251us) server_negotiation.cc:407] Connection header received
0924 10:00:43.457217 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:00:43.457223 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:00:43.457295 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:00:43.457385 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:00:43.458738 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:00:43.459767 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:00:43.461115 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:00:43.461744 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:00:43.461854 (+ 110us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:00:43.464566 (+ 2712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:00:43.464574 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:00:43.464578 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:00:43.464614 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:00:43.468269 (+ 3655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:00:43.469310 (+ 1041us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:00:43.469316 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:00:43.469321 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:00:43.469474 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:00:43.470015 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:00:43.470020 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:00:43.470024 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:00:43.470457 (+ 433us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:00:43.470728 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0924 10:00:43.472058 (+ 1330us) server_negotiation.cc:299] Negotiation successful
0924 10:00:43.472249 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260924 10:00:43.465500 20839 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd64f7a3058f46d8a3a42a96fb63798a [sys.catalog]: SysCatalogTable state changed. Reason: New leader dd64f7a3058f46d8a3a42a96fb63798a. Latest consensus state: current_term: 1 leader_uuid: "dd64f7a3058f46d8a3a42a96fb63798a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd64f7a3058f46d8a3a42a96fb63798a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44759 } } }
I20260924 10:00:43.472978 20839 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd64f7a3058f46d8a3a42a96fb63798a [sys.catalog]: This master's current role is: LEADER
I20260924 10:00:43.481789 20843 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:00:43.487784 20843 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:00:43.496256 20843 catalog_manager.cc:1347] Generated new cluster ID: 63fb8b98144c4cab9b93509a85e21790
I20260924 10:00:43.496364 20843 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:00:43.507025 20843 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:00:43.508275 20843 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:00:43.517726 20843 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dd64f7a3058f46d8a3a42a96fb63798a: Generated new TSK 0
I20260924 10:00:43.518544 20843 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 10:00:43.533532 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu pid 20760
I20260924 10:00:43.534682 20822 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:00:43.629159 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu with pid 20760
W20260924 10:00:43.643003 20745 connection.cc:535] client connection to 127.0.0.1:44759 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44759): Connection reset by peer (error 104)
2026-09-24T10:00:43Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1939 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790243735634835-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:00:43 dist-test-slave-w8d2 krb5kdc[20860](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:00:43 dist-test-slave-w8d2 krb5kdc[20860](info): set up 2 sockets
Sep 24 10:00:43 dist-test-slave-w8d2 krb5kdc[20860](info): commencing operation
krb5kdc: starting...
W20260924 10:00:44.643487 388 mini_kdc.cc:121] Time spent starting KDC: real 0.981s user 0.000s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790243735634835-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790243735634835-388-0/krb5kdc/test-user.keytab.
Sep 24 10:00:44 dist-test-slave-w8d2 krb5kdc[20860](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244044, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:00:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:00:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790243735634835-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790243735634835-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 10:00:44.872002 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
/tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790243735634835-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790243735634835-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790243735634835-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790243735634835-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46599
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790243735634835-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46599
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790243735634835-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790243735634835-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790243735634835-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:00:45.188009 20873 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:00:45.188308 20873 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:00:45.188442 20873 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:00:45.195499 20873 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:00:45.195603 20873 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:00:45.195678 20873 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:00:45.195756 20873 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:00:45.195813 20873 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:00:45.205473 20873 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790243735634835-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790243735634835-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46599
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790243735634835-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46599
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790243735634835-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790243735634835-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:00:45.206697 20873 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:00:45.207952 20873 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:00:45.218851 20878 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:00:45.218937 20879 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:00:45.219391 20881 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:00:45.219975 20873 server_base.cc:1034] running on GCE node
I20260924 10:00:45.220681 20873 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:00:45.222023 20873 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:00:45.223253 20873 hybrid_clock.cc:648] HybridClock initialized: now 1790244045223180 us; error 44 us; skew 500 ppm
I20260924 10:00:45.223593 20873 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:00:45 dist-test-slave-w8d2 krb5kdc[20860](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244045, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:00:45.226431 20873 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 10:00:45.227859 20873 webserver.cc:460] Webserver started at http://127.0.0.1:38127/ using document root <none> and password file <none>
I20260924 10:00:45.228519 20873 fs_manager.cc:359] Metadata directory not provided
I20260924 10:00:45.228628 20873 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:00:45.228901 20873 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:00:45.231855 20873 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790243735634835-388-0/minicluster-data/master-0/data/instance:
uuid: "47c33cb5e1fc42ceb9cb8a98d7b18fac"
format_stamp: "Formatted at 2026-09-24 10:00:45 on dist-test-slave-w8d2"
server_key: "cf95f860fc8415877c37a2fad36fee0e"
server_key_iv: "c312ea50a8d1f3c02abff4311a2377b4"
server_key_version: "encryptionkey@0"
I20260924 10:00:45.232605 20873 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790243735634835-388-0/minicluster-data/master-0/wal/instance:
uuid: "47c33cb5e1fc42ceb9cb8a98d7b18fac"
format_stamp: "Formatted at 2026-09-24 10:00:45 on dist-test-slave-w8d2"
server_key: "cf95f860fc8415877c37a2fad36fee0e"
server_key_iv: "c312ea50a8d1f3c02abff4311a2377b4"
server_key_version: "encryptionkey@0"
I20260924 10:00:45.237833 20873 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 10:00:45.241276 20888 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:00:45.242622 20873 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260924 10:00:45.242764 20873 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790243735634835-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790243735634835-388-0/minicluster-data/master-0/wal
uuid: "47c33cb5e1fc42ceb9cb8a98d7b18fac"
format_stamp: "Formatted at 2026-09-24 10:00:45 on dist-test-slave-w8d2"
server_key: "cf95f860fc8415877c37a2fad36fee0e"
server_key_iv: "c312ea50a8d1f3c02abff4311a2377b4"
server_key_version: "encryptionkey@0"
I20260924 10:00:45.242931 20873 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790243735634835-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790243735634835-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790243735634835-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:00:45.297683 20873 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:00:45.301916 20873 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:00:45.302166 20873 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:00:45.318873 20873 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46599
I20260924 10:00:45.318895 20939 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46599 every 8 connection(s)
I20260924 10:00:45.320176 20873 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790243735634835-388-0/minicluster-data/master-0/data/info.pb
I20260924 10:00:45.323791 20940 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:00:45.323808 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu as pid 20873
I20260924 10:00:45.323966 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790243735634835-388-0/minicluster-data/master-0/wal/instance
I20260924 10:00:45.324365 388 external_mini_cluster.cc:1428] Setting key e5bfd24ad6ae3fad561d88d0f945c424
I20260924 10:00:45.339293 20940 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 47c33cb5e1fc42ceb9cb8a98d7b18fac: Bootstrap starting.
I20260924 10:00:45.339484 20943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:45.326128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59587 (local address 127.0.0.1:46599)
0924 10:00:45.326932 (+ 804us) server_negotiation.cc:206] Beginning negotiation
0924 10:00:45.326948 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 10:00:45.327002 (+ 54us) server_negotiation.cc:407] Connection header received
0924 10:00:45.328851 (+ 1849us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:00:45.328881 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:00:45.329460 (+ 579us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:00:45.329929 (+ 469us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:00:45.330488 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:00:45.332490 (+ 2002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:00:45.333836 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:00:45.334906 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:00:45.335158 (+ 252us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:00:45.336514 (+ 1356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:00:45.336529 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:00:45.336556 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:00:45.337352 (+ 796us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:00:45.337459 (+ 107us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:00:45.337493 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:00:45.337651 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0924 10:00:45.337877 (+ 226us) server_negotiation.cc:299] Negotiation successful
0924 10:00:45.338074 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":555,"thread_start_us":333,"threads_started":1}
I20260924 10:00:45.342697 20940 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 47c33cb5e1fc42ceb9cb8a98d7b18fac: Neither blocks nor log segments found. Creating new log.
I20260924 10:00:45.343323 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 10:00:45.343880 20940 log.cc:824] T 00000000000000000000000000000000 P 47c33cb5e1fc42ceb9cb8a98d7b18fac: Log is configured to *not* fsync() on all Append() calls
I20260924 10:00:45.347194 20940 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 47c33cb5e1fc42ceb9cb8a98d7b18fac: No bootstrap required, opened a new log
I20260924 10:00:45.354614 20943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:45.347916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51509 (local address 127.0.0.1:46599)
0924 10:00:45.348099 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0924 10:00:45.348106 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:00:45.348604 (+ 498us) server_negotiation.cc:407] Connection header received
0924 10:00:45.348814 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:00:45.348819 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:00:45.348880 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:00:45.348958 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:00:45.349474 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:00:45.350251 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:00:45.352119 (+ 1868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:00:45.352611 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:00:45.352708 (+ 97us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:00:45.353897 (+ 1189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:00:45.353902 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:00:45.353905 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:00:45.353922 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:00:45.353967 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:00:45.353982 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:00:45.354059 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 10:00:45.354238 (+ 179us) server_negotiation.cc:299] Negotiation successful
0924 10:00:45.354287 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 10:00:45.356783 20940 raft_consensus.cc:348] T 00000000000000000000000000000000 P 47c33cb5e1fc42ceb9cb8a98d7b18fac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47c33cb5e1fc42ceb9cb8a98d7b18fac" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46599 } }
I20260924 10:00:45.357146 20940 raft_consensus.cc:374] T 00000000000000000000000000000000 P 47c33cb5e1fc42ceb9cb8a98d7b18fac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:00:45.357240 20940 raft_consensus.cc:729] T 00000000000000000000000000000000 P 47c33cb5e1fc42ceb9cb8a98d7b18fac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47c33cb5e1fc42ceb9cb8a98d7b18fac, State: Initialized, Role: FOLLOWER
I20260924 10:00:45.357872 20940 consensus_queue.cc:260] T 00000000000000000000000000000000 P 47c33cb5e1fc42ceb9cb8a98d7b18fac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47c33cb5e1fc42ceb9cb8a98d7b18fac" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46599 } }
I20260924 10:00:45.358132 20940 raft_consensus.cc:388] T 00000000000000000000000000000000 P 47c33cb5e1fc42ceb9cb8a98d7b18fac [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:00:45.358261 20940 raft_consensus.cc:482] T 00000000000000000000000000000000 P 47c33cb5e1fc42ceb9cb8a98d7b18fac [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:00:45.358419 20940 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 47c33cb5e1fc42ceb9cb8a98d7b18fac [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:00:45.360639 20940 raft_consensus.cc:504] T 00000000000000000000000000000000 P 47c33cb5e1fc42ceb9cb8a98d7b18fac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47c33cb5e1fc42ceb9cb8a98d7b18fac" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46599 } }
I20260924 10:00:45.361150 20940 leader_election.cc:302] T 00000000000000000000000000000000 P 47c33cb5e1fc42ceb9cb8a98d7b18fac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 47c33cb5e1fc42ceb9cb8a98d7b18fac; no voters:
I20260924 10:00:45.361716 20940 leader_election.cc:288] T 00000000000000000000000000000000 P 47c33cb5e1fc42ceb9cb8a98d7b18fac [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:00:45.361857 20950 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 47c33cb5e1fc42ceb9cb8a98d7b18fac [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:00:45.362954 20950 raft_consensus.cc:686] T 00000000000000000000000000000000 P 47c33cb5e1fc42ceb9cb8a98d7b18fac [term 1 LEADER]: Becoming Leader. State: Replica: 47c33cb5e1fc42ceb9cb8a98d7b18fac, State: Running, Role: LEADER
I20260924 10:00:45.363498 20950 consensus_queue.cc:237] T 00000000000000000000000000000000 P 47c33cb5e1fc42ceb9cb8a98d7b18fac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47c33cb5e1fc42ceb9cb8a98d7b18fac" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46599 } }
I20260924 10:00:45.363801 20940 sys_catalog.cc:564] T 00000000000000000000000000000000 P 47c33cb5e1fc42ceb9cb8a98d7b18fac [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:00:45.365793 20952 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47c33cb5e1fc42ceb9cb8a98d7b18fac [sys.catalog]: SysCatalogTable state changed. Reason: New leader 47c33cb5e1fc42ceb9cb8a98d7b18fac. Latest consensus state: current_term: 1 leader_uuid: "47c33cb5e1fc42ceb9cb8a98d7b18fac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47c33cb5e1fc42ceb9cb8a98d7b18fac" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46599 } } }
I20260924 10:00:45.366077 20952 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47c33cb5e1fc42ceb9cb8a98d7b18fac [sys.catalog]: This master's current role is: LEADER
I20260924 10:00:45.366577 20951 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47c33cb5e1fc42ceb9cb8a98d7b18fac [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "47c33cb5e1fc42ceb9cb8a98d7b18fac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47c33cb5e1fc42ceb9cb8a98d7b18fac" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46599 } } }
I20260924 10:00:45.366855 20951 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47c33cb5e1fc42ceb9cb8a98d7b18fac [sys.catalog]: This master's current role is: LEADER
I20260924 10:00:45.366879 20956 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:00:45.378237 20956 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:00:45.385823 20956 catalog_manager.cc:1347] Generated new cluster ID: b7b8fc7b3d734e25b30e1725ab266f64
I20260924 10:00:45.385942 20956 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:00:45.421164 20956 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:00:45.422459 20956 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:00:45.432948 20956 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 47c33cb5e1fc42ceb9cb8a98d7b18fac: Generated new TSK 0
I20260924 10:00:45.434051 20956 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 10:00:45.449541 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu pid 20873
W20260924 10:00:45.450955 20935 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:59587
W20260924 10:00:45.451639 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu pid 20873: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 10:00:45.451843 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu with pid 20873
W20260924 10:00:45.465457 20858 connection.cc:535] client connection to 127.0.0.1:46599 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46599): Connection reset by peer (error 104)
2026-09-24T10:00:45Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1821 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790243735634835-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:00:45 dist-test-slave-w8d2 krb5kdc[20971](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:00:45 dist-test-slave-w8d2 krb5kdc[20971](info): set up 2 sockets
Sep 24 10:00:45 dist-test-slave-w8d2 krb5kdc[20971](info): commencing operation
krb5kdc: starting...
W20260924 10:00:46.455458 388 mini_kdc.cc:121] Time spent starting KDC: real 0.972s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790243735634835-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790243735634835-388-0/krb5kdc/test-user.keytab.
Sep 24 10:00:46 dist-test-slave-w8d2 krb5kdc[20971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244046, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:00:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:00:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790243735634835-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790243735634835-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 10:00:46.680332 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
/tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790243735634835-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790243735634835-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790243735634835-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790243735634835-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43555
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790243735634835-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43555
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790243735634835-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790243735634835-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790243735634835-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:00:46.993188 20984 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:00:46.993487 20984 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:00:46.993593 20984 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:00:47.000890 20984 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:00:47.001011 20984 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:00:47.001065 20984 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:00:47.001123 20984 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:00:47.001166 20984 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:00:47.010998 20984 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790243735634835-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790243735634835-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43555
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790243735634835-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43555
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790243735634835-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790243735634835-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:00:47.012236 20984 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:00:47.013451 20984 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:00:47.024709 20989 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:00:47.024744 20990 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:00:47.024802 20992 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:00:47.025746 20984 server_base.cc:1034] running on GCE node
I20260924 10:00:47.026543 20984 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:00:47.028052 20984 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:00:47.029284 20984 hybrid_clock.cc:648] HybridClock initialized: now 1790244047029230 us; error 36 us; skew 500 ppm
I20260924 10:00:47.029706 20984 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:00:47 dist-test-slave-w8d2 krb5kdc[20971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244047, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:00:47.036000 20984 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 10:00:47.037693 20984 webserver.cc:460] Webserver started at http://127.0.0.1:33147/ using document root <none> and password file <none>
I20260924 10:00:47.038370 20984 fs_manager.cc:359] Metadata directory not provided
I20260924 10:00:47.038480 20984 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:00:47.038810 20984 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:00:47.041857 20984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790243735634835-388-0/minicluster-data/master-0/data/instance:
uuid: "fbdd3d9777c34584a52d72c75de1b190"
format_stamp: "Formatted at 2026-09-24 10:00:47 on dist-test-slave-w8d2"
server_key: "b0d74c27258f0ffa73a0db4efefbfd17"
server_key_iv: "696c404d9b27abe6312283dcacf7418c"
server_key_version: "encryptionkey@0"
I20260924 10:00:47.042631 20984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790243735634835-388-0/minicluster-data/master-0/wal/instance:
uuid: "fbdd3d9777c34584a52d72c75de1b190"
format_stamp: "Formatted at 2026-09-24 10:00:47 on dist-test-slave-w8d2"
server_key: "b0d74c27258f0ffa73a0db4efefbfd17"
server_key_iv: "696c404d9b27abe6312283dcacf7418c"
server_key_version: "encryptionkey@0"
I20260924 10:00:47.047765 20984 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260924 10:00:47.051133 20999 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:00:47.052476 20984 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 10:00:47.052635 20984 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790243735634835-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790243735634835-388-0/minicluster-data/master-0/wal
uuid: "fbdd3d9777c34584a52d72c75de1b190"
format_stamp: "Formatted at 2026-09-24 10:00:47 on dist-test-slave-w8d2"
server_key: "b0d74c27258f0ffa73a0db4efefbfd17"
server_key_iv: "696c404d9b27abe6312283dcacf7418c"
server_key_version: "encryptionkey@0"
I20260924 10:00:47.052814 20984 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790243735634835-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790243735634835-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790243735634835-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:00:47.058456 20984 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:00:47.062178 20984 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:00:47.062433 20984 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:00:47.079772 20984 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43555
I20260924 10:00:47.079793 21050 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43555 every 8 connection(s)
I20260924 10:00:47.081099 20984 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790243735634835-388-0/minicluster-data/master-0/data/info.pb
I20260924 10:00:47.084646 21051 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:00:47.089864 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu as pid 20984
I20260924 10:00:47.090025 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790243735634835-388-0/minicluster-data/master-0/wal/instance
I20260924 10:00:47.090401 388 external_mini_cluster.cc:1428] Setting key 9afd660d0fa525d0598af164d4d1d73d
I20260924 10:00:47.094765 21051 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fbdd3d9777c34584a52d72c75de1b190: Bootstrap starting.
I20260924 10:00:47.097533 21054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:47.091982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34807 (local address 127.0.0.1:43555)
0924 10:00:47.092770 (+ 788us) server_negotiation.cc:206] Beginning negotiation
0924 10:00:47.092785 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 10:00:47.092827 (+ 42us) server_negotiation.cc:407] Connection header received
0924 10:00:47.094431 (+ 1604us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:00:47.094453 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:00:47.094782 (+ 329us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:00:47.095166 (+ 384us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:00:47.096114 (+ 948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:00:47.096119 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:00:47.096131 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:00:47.096144 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:00:47.096197 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:00:47.096218 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:00:47.096385 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0924 10:00:47.096483 (+ 98us) server_negotiation.cc:299] Negotiation successful
0924 10:00:47.096682 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":567,"thread_start_us":356,"threads_started":1}
I20260924 10:00:47.098209 21051 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fbdd3d9777c34584a52d72c75de1b190: Neither blocks nor log segments found. Creating new log.
I20260924 10:00:47.099334 21051 log.cc:824] T 00000000000000000000000000000000 P fbdd3d9777c34584a52d72c75de1b190: Log is configured to *not* fsync() on all Append() calls
I20260924 10:00:47.101109 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 10:00:47.102203 21051 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fbdd3d9777c34584a52d72c75de1b190: No bootstrap required, opened a new log
I20260924 10:00:47.108232 21054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:47.106010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32867 (local address 127.0.0.1:43555)
0924 10:00:47.106204 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0924 10:00:47.106210 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:00:47.106462 (+ 252us) server_negotiation.cc:407] Connection header received
0924 10:00:47.106614 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:00:47.106617 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:00:47.106662 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:00:47.106732 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:00:47.107671 (+ 939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:00:47.107675 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:00:47.107678 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:00:47.107692 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:00:47.107731 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:00:47.107744 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:00:47.107837 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0924 10:00:47.107934 (+ 97us) server_negotiation.cc:299] Negotiation successful
0924 10:00:47.107979 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260924 10:00:47.111263 21051 raft_consensus.cc:348] T 00000000000000000000000000000000 P fbdd3d9777c34584a52d72c75de1b190 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fbdd3d9777c34584a52d72c75de1b190" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43555 } }
I20260924 10:00:47.111626 21051 raft_consensus.cc:374] T 00000000000000000000000000000000 P fbdd3d9777c34584a52d72c75de1b190 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:00:47.111721 21051 raft_consensus.cc:729] T 00000000000000000000000000000000 P fbdd3d9777c34584a52d72c75de1b190 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbdd3d9777c34584a52d72c75de1b190, State: Initialized, Role: FOLLOWER
I20260924 10:00:47.112274 21051 consensus_queue.cc:260] T 00000000000000000000000000000000 P fbdd3d9777c34584a52d72c75de1b190 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fbdd3d9777c34584a52d72c75de1b190" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43555 } }
I20260924 10:00:47.112486 21051 raft_consensus.cc:388] T 00000000000000000000000000000000 P fbdd3d9777c34584a52d72c75de1b190 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:00:47.112617 21051 raft_consensus.cc:482] T 00000000000000000000000000000000 P fbdd3d9777c34584a52d72c75de1b190 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:00:47.112767 21051 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fbdd3d9777c34584a52d72c75de1b190 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:00:47.115068 21051 raft_consensus.cc:504] T 00000000000000000000000000000000 P fbdd3d9777c34584a52d72c75de1b190 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fbdd3d9777c34584a52d72c75de1b190" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43555 } }
I20260924 10:00:47.115554 21051 leader_election.cc:302] T 00000000000000000000000000000000 P fbdd3d9777c34584a52d72c75de1b190 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fbdd3d9777c34584a52d72c75de1b190; no voters:
I20260924 10:00:47.116075 21051 leader_election.cc:288] T 00000000000000000000000000000000 P fbdd3d9777c34584a52d72c75de1b190 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:00:47.116206 21061 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fbdd3d9777c34584a52d72c75de1b190 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:00:47.116781 21061 raft_consensus.cc:686] T 00000000000000000000000000000000 P fbdd3d9777c34584a52d72c75de1b190 [term 1 LEADER]: Becoming Leader. State: Replica: fbdd3d9777c34584a52d72c75de1b190, State: Running, Role: LEADER
I20260924 10:00:47.117261 21061 consensus_queue.cc:237] T 00000000000000000000000000000000 P fbdd3d9777c34584a52d72c75de1b190 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fbdd3d9777c34584a52d72c75de1b190" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43555 } }
I20260924 10:00:47.117954 21051 sys_catalog.cc:564] T 00000000000000000000000000000000 P fbdd3d9777c34584a52d72c75de1b190 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:00:47.119819 21062 sys_catalog.cc:455] T 00000000000000000000000000000000 P fbdd3d9777c34584a52d72c75de1b190 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fbdd3d9777c34584a52d72c75de1b190" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fbdd3d9777c34584a52d72c75de1b190" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43555 } } }
I20260924 10:00:47.120318 21062 sys_catalog.cc:458] T 00000000000000000000000000000000 P fbdd3d9777c34584a52d72c75de1b190 [sys.catalog]: This master's current role is: LEADER
I20260924 10:00:47.120435 21063 sys_catalog.cc:455] T 00000000000000000000000000000000 P fbdd3d9777c34584a52d72c75de1b190 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fbdd3d9777c34584a52d72c75de1b190. Latest consensus state: current_term: 1 leader_uuid: "fbdd3d9777c34584a52d72c75de1b190" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fbdd3d9777c34584a52d72c75de1b190" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43555 } } }
I20260924 10:00:47.120787 21063 sys_catalog.cc:458] T 00000000000000000000000000000000 P fbdd3d9777c34584a52d72c75de1b190 [sys.catalog]: This master's current role is: LEADER
I20260924 10:00:47.121207 21068 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:00:47.126552 21068 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:00:47.134234 21068 catalog_manager.cc:1347] Generated new cluster ID: a757ec198e4847bfa3b8f940aac2dbad
I20260924 10:00:47.134335 21068 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:00:47.147747 21068 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:00:47.148104 21068 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:00:47.159210 21068 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fbdd3d9777c34584a52d72c75de1b190: Generated new TSK 0
I20260924 10:00:47.160000 21068 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 10:00:47.168658 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu pid 20984
W20260924 10:00:47.169719 21046 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:34807
W20260924 10:00:47.170321 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu pid 20984: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 10:00:47.170441 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu with pid 20984
2026-09-24T10:00:47Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1718 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790243735634835-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:00:47 dist-test-slave-w8d2 krb5kdc[21082](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:00:47 dist-test-slave-w8d2 krb5kdc[21082](info): set up 2 sockets
Sep 24 10:00:47 dist-test-slave-w8d2 krb5kdc[21082](info): commencing operation
krb5kdc: starting...
W20260924 10:00:48.171401 388 mini_kdc.cc:121] Time spent starting KDC: real 0.970s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790243735634835-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790243735634835-388-0/krb5kdc/test-user.keytab.
Sep 24 10:00:48 dist-test-slave-w8d2 krb5kdc[21082](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244048, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:00:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:00:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790243735634835-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790243735634835-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 10:00:48.402379 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
/tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790243735634835-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790243735634835-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790243735634835-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790243735634835-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39705
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790243735634835-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39705
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790243735634835-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790243735634835-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790243735634835-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:00:48.718480 21095 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:00:48.718743 21095 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:00:48.718889 21095 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:00:48.725903 21095 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:00:48.726001 21095 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:00:48.726053 21095 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:00:48.726133 21095 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:00:48.726188 21095 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:00:48.735733 21095 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790243735634835-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790243735634835-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39705
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790243735634835-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39705
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790243735634835-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790243735634835-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:00:48.737280 21095 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:00:48.738583 21095 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:00:48.751025 21101 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:00:48.751053 21100 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:00:48.751557 21103 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:00:48.752171 21095 server_base.cc:1034] running on GCE node
I20260924 10:00:48.752880 21095 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:00:48.754215 21095 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:00:48.755424 21095 hybrid_clock.cc:648] HybridClock initialized: now 1790244048755376 us; error 33 us; skew 500 ppm
I20260924 10:00:48.755751 21095 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:00:48 dist-test-slave-w8d2 krb5kdc[21082](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244048, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:00:48.758637 21095 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 10:00:48.760107 21095 webserver.cc:460] Webserver started at http://127.0.0.1:34837/ using document root <none> and password file <none>
I20260924 10:00:48.760725 21095 fs_manager.cc:359] Metadata directory not provided
I20260924 10:00:48.760828 21095 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:00:48.761070 21095 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:00:48.763945 21095 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790243735634835-388-0/minicluster-data/master-0/data/instance:
uuid: "3c8a945f659648f7be7381d2750da6d7"
format_stamp: "Formatted at 2026-09-24 10:00:48 on dist-test-slave-w8d2"
server_key: "32c3779b81ea7a240792113eebe6cc3d"
server_key_iv: "a969bfb1d653d409b89054f603b0b850"
server_key_version: "encryptionkey@0"
I20260924 10:00:48.764617 21095 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790243735634835-388-0/minicluster-data/master-0/wal/instance:
uuid: "3c8a945f659648f7be7381d2750da6d7"
format_stamp: "Formatted at 2026-09-24 10:00:48 on dist-test-slave-w8d2"
server_key: "32c3779b81ea7a240792113eebe6cc3d"
server_key_iv: "a969bfb1d653d409b89054f603b0b850"
server_key_version: "encryptionkey@0"
I20260924 10:00:48.769503 21095 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 10:00:48.772771 21110 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:00:48.774039 21095 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260924 10:00:48.774173 21095 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790243735634835-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790243735634835-388-0/minicluster-data/master-0/wal
uuid: "3c8a945f659648f7be7381d2750da6d7"
format_stamp: "Formatted at 2026-09-24 10:00:48 on dist-test-slave-w8d2"
server_key: "32c3779b81ea7a240792113eebe6cc3d"
server_key_iv: "a969bfb1d653d409b89054f603b0b850"
server_key_version: "encryptionkey@0"
I20260924 10:00:48.774339 21095 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790243735634835-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790243735634835-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790243735634835-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:00:48.787719 21095 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:00:48.791154 21095 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:00:48.791414 21095 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:00:48.807364 21095 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39705
I20260924 10:00:48.807365 21161 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39705 every 8 connection(s)
I20260924 10:00:48.808643 21095 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790243735634835-388-0/minicluster-data/master-0/data/info.pb
I20260924 10:00:48.812057 21162 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:00:48.813513 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu as pid 21095
I20260924 10:00:48.813678 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790243735634835-388-0/minicluster-data/master-0/wal/instance
I20260924 10:00:48.814127 388 external_mini_cluster.cc:1428] Setting key 18e95db1abc0500e2db83b14c1cce617
I20260924 10:00:48.825009 21162 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c8a945f659648f7be7381d2750da6d7: Bootstrap starting.
Sep 24 10:00:48 dist-test-slave-w8d2 krb5kdc[21082](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244048, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 10:00:48.828423 21162 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3c8a945f659648f7be7381d2750da6d7: Neither blocks nor log segments found. Creating new log.
I20260924 10:00:48.829476 21162 log.cc:824] T 00000000000000000000000000000000 P 3c8a945f659648f7be7381d2750da6d7: Log is configured to *not* fsync() on all Append() calls
I20260924 10:00:48.831995 21162 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c8a945f659648f7be7381d2750da6d7: No bootstrap required, opened a new log
I20260924 10:00:48.835716 21165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:48.815727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50757 (local address 127.0.0.1:39705)
0924 10:00:48.816484 (+ 757us) server_negotiation.cc:206] Beginning negotiation
0924 10:00:48.816498 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 10:00:48.816538 (+ 40us) server_negotiation.cc:407] Connection header received
0924 10:00:48.818087 (+ 1549us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:00:48.818110 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:00:48.818491 (+ 381us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:00:48.818790 (+ 299us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:00:48.820429 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:00:48.821678 (+ 1249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:00:48.823424 (+ 1746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:00:48.824052 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:00:48.824172 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:00:48.828608 (+ 4436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:00:48.828616 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:00:48.828629 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:00:48.828665 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:00:48.831544 (+ 2879us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:00:48.832173 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:00:48.832181 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:00:48.832188 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:00:48.832383 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:00:48.832741 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:00:48.832746 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:00:48.832749 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:00:48.833407 (+ 658us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:00:48.833695 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0924 10:00:48.834023 (+ 328us) server_negotiation.cc:299] Negotiation successful
0924 10:00:48.834435 (+ 412us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":557,"thread_start_us":339,"threads_started":1}
I20260924 10:00:48.839115 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 10:00:48.840888 21162 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3c8a945f659648f7be7381d2750da6d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c8a945f659648f7be7381d2750da6d7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39705 } }
I20260924 10:00:48.841282 21162 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3c8a945f659648f7be7381d2750da6d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:00:48.841373 21162 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3c8a945f659648f7be7381d2750da6d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c8a945f659648f7be7381d2750da6d7, State: Initialized, Role: FOLLOWER
I20260924 10:00:48.841979 21162 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3c8a945f659648f7be7381d2750da6d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c8a945f659648f7be7381d2750da6d7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39705 } }
I20260924 10:00:48.842253 21162 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3c8a945f659648f7be7381d2750da6d7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:00:48.842396 21162 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3c8a945f659648f7be7381d2750da6d7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:00:48.842615 21162 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3c8a945f659648f7be7381d2750da6d7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:00:48.845268 21162 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3c8a945f659648f7be7381d2750da6d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c8a945f659648f7be7381d2750da6d7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39705 } }
I20260924 10:00:48.845703 21162 leader_election.cc:302] T 00000000000000000000000000000000 P 3c8a945f659648f7be7381d2750da6d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3c8a945f659648f7be7381d2750da6d7; no voters:
I20260924 10:00:48.846249 21162 leader_election.cc:288] T 00000000000000000000000000000000 P 3c8a945f659648f7be7381d2750da6d7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:00:48.846357 21172 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3c8a945f659648f7be7381d2750da6d7 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:00:48.848416 21162 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3c8a945f659648f7be7381d2750da6d7 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:00:48.852123 21172 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3c8a945f659648f7be7381d2750da6d7 [term 1 LEADER]: Becoming Leader. State: Replica: 3c8a945f659648f7be7381d2750da6d7, State: Running, Role: LEADER
I20260924 10:00:48.852769 21172 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3c8a945f659648f7be7381d2750da6d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c8a945f659648f7be7381d2750da6d7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39705 } }
I20260924 10:00:48.855684 21173 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c8a945f659648f7be7381d2750da6d7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3c8a945f659648f7be7381d2750da6d7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c8a945f659648f7be7381d2750da6d7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39705 } } }
I20260924 10:00:48.856050 21173 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c8a945f659648f7be7381d2750da6d7 [sys.catalog]: This master's current role is: LEADER
I20260924 10:00:48.856483 21174 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c8a945f659648f7be7381d2750da6d7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3c8a945f659648f7be7381d2750da6d7. Latest consensus state: current_term: 1 leader_uuid: "3c8a945f659648f7be7381d2750da6d7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c8a945f659648f7be7381d2750da6d7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39705 } } }
I20260924 10:00:48.856776 21174 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c8a945f659648f7be7381d2750da6d7 [sys.catalog]: This master's current role is: LEADER
I20260924 10:00:48.857811 21165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:48.843981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58119 (local address 127.0.0.1:39705)
0924 10:00:48.844151 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0924 10:00:48.844156 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:00:48.844514 (+ 358us) server_negotiation.cc:407] Connection header received
0924 10:00:48.844626 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:00:48.844631 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:00:48.844700 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:00:48.844793 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:00:48.846227 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:00:48.847333 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:00:48.848571 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:00:48.849019 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:00:48.849127 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:00:48.851352 (+ 2225us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:00:48.851359 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:00:48.851363 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:00:48.851402 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:00:48.854018 (+ 2616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:00:48.854775 (+ 757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:00:48.854780 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:00:48.854784 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:00:48.854926 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:00:48.855413 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:00:48.855418 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:00:48.855421 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:00:48.855858 (+ 437us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:00:48.856102 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0924 10:00:48.857188 (+ 1086us) server_negotiation.cc:299] Negotiation successful
0924 10:00:48.857438 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 10:00:48.864104 21179 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:00:48.869374 21179 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:00:48.878396 21179 catalog_manager.cc:1347] Generated new cluster ID: 2ac6120bc85b4b918a7010b6e94506d4
I20260924 10:00:48.878499 21179 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:00:48.891342 21179 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:00:48.892731 21179 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:00:48.901760 21179 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3c8a945f659648f7be7381d2750da6d7: Generated new TSK 0
I20260924 10:00:48.902500 21179 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 10:00:48.911334 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu pid 21095
I20260924 10:00:48.912281 21157 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:00:49.000389 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu with pid 21095
2026-09-24T10:00:49Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1830 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790243735634835-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:00:49 dist-test-slave-w8d2 krb5kdc[21195](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:00:49 dist-test-slave-w8d2 krb5kdc[21195](info): set up 2 sockets
Sep 24 10:00:49 dist-test-slave-w8d2 krb5kdc[21195](info): commencing operation
krb5kdc: starting...
W20260924 10:00:50.003454 388 mini_kdc.cc:121] Time spent starting KDC: real 0.971s user 0.006s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790243735634835-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790243735634835-388-0/krb5kdc/test-user.keytab.
Sep 24 10:00:50 dist-test-slave-w8d2 krb5kdc[21195](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244050, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:00:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:00:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790243735634835-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790243735634835-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 10:00:50.229861 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
/tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790243735634835-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790243735634835-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790243735634835-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790243735634835-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46405
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790243735634835-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46405
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790243735634835-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790243735634835-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790243735634835-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:00:50.546864 21208 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:00:50.547158 21208 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:00:50.547307 21208 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:00:50.554535 21208 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:00:50.554666 21208 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:00:50.554744 21208 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:00:50.554821 21208 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:00:50.554884 21208 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:00:50.564870 21208 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790243735634835-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790243735634835-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46405
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790243735634835-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46405
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790243735634835-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790243735634835-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:00:50.566227 21208 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:00:50.567544 21208 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:00:50.578756 21214 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:00:50.578820 21213 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:00:50.578820 21216 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:00:50.579572 21208 server_base.cc:1034] running on GCE node
I20260924 10:00:50.580385 21208 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:00:50.581837 21208 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:00:50.583048 21208 hybrid_clock.cc:648] HybridClock initialized: now 1790244050582986 us; error 52 us; skew 500 ppm
I20260924 10:00:50.583424 21208 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:00:50 dist-test-slave-w8d2 krb5kdc[21195](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244050, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:00:50.586365 21208 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 10:00:50.587811 21208 webserver.cc:460] Webserver started at http://127.0.0.1:39303/ using document root <none> and password file <none>
I20260924 10:00:50.588475 21208 fs_manager.cc:359] Metadata directory not provided
I20260924 10:00:50.588585 21208 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:00:50.588860 21208 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:00:50.591799 21208 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790243735634835-388-0/minicluster-data/master-0/data/instance:
uuid: "d0c6f777fd85462bb4da4d02642c3dc9"
format_stamp: "Formatted at 2026-09-24 10:00:50 on dist-test-slave-w8d2"
server_key: "edec1354a6a08adbd33ed14701957374"
server_key_iv: "57880701bb0e8a6edc88a69d9d718c34"
server_key_version: "encryptionkey@0"
I20260924 10:00:50.592522 21208 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790243735634835-388-0/minicluster-data/master-0/wal/instance:
uuid: "d0c6f777fd85462bb4da4d02642c3dc9"
format_stamp: "Formatted at 2026-09-24 10:00:50 on dist-test-slave-w8d2"
server_key: "edec1354a6a08adbd33ed14701957374"
server_key_iv: "57880701bb0e8a6edc88a69d9d718c34"
server_key_version: "encryptionkey@0"
I20260924 10:00:50.597422 21208 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260924 10:00:50.600723 21223 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:00:50.602039 21208 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 10:00:50.602213 21208 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790243735634835-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790243735634835-388-0/minicluster-data/master-0/wal
uuid: "d0c6f777fd85462bb4da4d02642c3dc9"
format_stamp: "Formatted at 2026-09-24 10:00:50 on dist-test-slave-w8d2"
server_key: "edec1354a6a08adbd33ed14701957374"
server_key_iv: "57880701bb0e8a6edc88a69d9d718c34"
server_key_version: "encryptionkey@0"
I20260924 10:00:50.602389 21208 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790243735634835-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790243735634835-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790243735634835-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:00:50.666333 21208 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:00:50.670660 21208 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:00:50.671056 21208 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:00:50.688066 21208 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46405
I20260924 10:00:50.688090 21274 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46405 every 8 connection(s)
I20260924 10:00:50.689440 21208 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790243735634835-388-0/minicluster-data/master-0/data/info.pb
I20260924 10:00:50.693178 21275 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:00:50.693373 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu as pid 21208
I20260924 10:00:50.693526 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790243735634835-388-0/minicluster-data/master-0/wal/instance
I20260924 10:00:50.693931 388 external_mini_cluster.cc:1428] Setting key c7c6397e8c8aa0f1f914fb6d2bbf595e
I20260924 10:00:50.707685 21275 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0c6f777fd85462bb4da4d02642c3dc9: Bootstrap starting.
I20260924 10:00:50.707959 21278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:50.695681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36723 (local address 127.0.0.1:46405)
0924 10:00:50.696519 (+ 838us) server_negotiation.cc:206] Beginning negotiation
0924 10:00:50.696536 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0924 10:00:50.696592 (+ 56us) server_negotiation.cc:407] Connection header received
0924 10:00:50.698271 (+ 1679us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:00:50.698300 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:00:50.698862 (+ 562us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:00:50.699384 (+ 522us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:00:50.699924 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:00:50.701894 (+ 1970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:00:50.703264 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:00:50.704310 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:00:50.704510 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:00:50.705565 (+ 1055us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:00:50.705573 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:00:50.705588 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:00:50.706163 (+ 575us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:00:50.706242 (+ 79us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:00:50.706268 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:00:50.706351 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0924 10:00:50.706616 (+ 265us) server_negotiation.cc:299] Negotiation successful
0924 10:00:50.706927 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":580,"thread_start_us":346,"threads_started":1}
I20260924 10:00:50.710827 21275 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d0c6f777fd85462bb4da4d02642c3dc9: Neither blocks nor log segments found. Creating new log.
I20260924 10:00:50.711735 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 10:00:50.711972 21275 log.cc:824] T 00000000000000000000000000000000 P d0c6f777fd85462bb4da4d02642c3dc9: Log is configured to *not* fsync() on all Append() calls
I20260924 10:00:50.715138 21275 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0c6f777fd85462bb4da4d02642c3dc9: No bootstrap required, opened a new log
I20260924 10:00:50.723223 21278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:50.716615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44561 (local address 127.0.0.1:46405)
0924 10:00:50.716791 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0924 10:00:50.716797 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:00:50.717124 (+ 327us) server_negotiation.cc:407] Connection header received
0924 10:00:50.717314 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:00:50.717317 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:00:50.717377 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:00:50.717456 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:00:50.718063 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:00:50.718831 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:00:50.720748 (+ 1917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:00:50.721217 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:00:50.721318 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:00:50.722505 (+ 1187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:00:50.722509 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:00:50.722512 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:00:50.722534 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:00:50.722575 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:00:50.722590 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:00:50.722664 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 10:00:50.722816 (+ 152us) server_negotiation.cc:299] Negotiation successful
0924 10:00:50.722899 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260924 10:00:50.725001 21275 raft_consensus.cc:348] T 00000000000000000000000000000000 P d0c6f777fd85462bb4da4d02642c3dc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0c6f777fd85462bb4da4d02642c3dc9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46405 } }
I20260924 10:00:50.725394 21275 raft_consensus.cc:374] T 00000000000000000000000000000000 P d0c6f777fd85462bb4da4d02642c3dc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:00:50.725487 21275 raft_consensus.cc:729] T 00000000000000000000000000000000 P d0c6f777fd85462bb4da4d02642c3dc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0c6f777fd85462bb4da4d02642c3dc9, State: Initialized, Role: FOLLOWER
I20260924 10:00:50.726028 21275 consensus_queue.cc:260] T 00000000000000000000000000000000 P d0c6f777fd85462bb4da4d02642c3dc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0c6f777fd85462bb4da4d02642c3dc9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46405 } }
I20260924 10:00:50.726308 21275 raft_consensus.cc:388] T 00000000000000000000000000000000 P d0c6f777fd85462bb4da4d02642c3dc9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:00:50.726433 21275 raft_consensus.cc:482] T 00000000000000000000000000000000 P d0c6f777fd85462bb4da4d02642c3dc9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:00:50.726581 21275 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d0c6f777fd85462bb4da4d02642c3dc9 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:00:50.728763 21275 raft_consensus.cc:504] T 00000000000000000000000000000000 P d0c6f777fd85462bb4da4d02642c3dc9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0c6f777fd85462bb4da4d02642c3dc9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46405 } }
I20260924 10:00:50.729255 21275 leader_election.cc:302] T 00000000000000000000000000000000 P d0c6f777fd85462bb4da4d02642c3dc9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d0c6f777fd85462bb4da4d02642c3dc9; no voters:
I20260924 10:00:50.729779 21275 leader_election.cc:288] T 00000000000000000000000000000000 P d0c6f777fd85462bb4da4d02642c3dc9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:00:50.729894 21285 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d0c6f777fd85462bb4da4d02642c3dc9 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:00:50.731118 21285 raft_consensus.cc:686] T 00000000000000000000000000000000 P d0c6f777fd85462bb4da4d02642c3dc9 [term 1 LEADER]: Becoming Leader. State: Replica: d0c6f777fd85462bb4da4d02642c3dc9, State: Running, Role: LEADER
I20260924 10:00:50.731707 21285 consensus_queue.cc:237] T 00000000000000000000000000000000 P d0c6f777fd85462bb4da4d02642c3dc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0c6f777fd85462bb4da4d02642c3dc9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46405 } }
I20260924 10:00:50.732350 21275 sys_catalog.cc:564] T 00000000000000000000000000000000 P d0c6f777fd85462bb4da4d02642c3dc9 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:00:50.734155 21286 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0c6f777fd85462bb4da4d02642c3dc9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d0c6f777fd85462bb4da4d02642c3dc9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0c6f777fd85462bb4da4d02642c3dc9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46405 } } }
I20260924 10:00:50.734599 21286 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0c6f777fd85462bb4da4d02642c3dc9 [sys.catalog]: This master's current role is: LEADER
I20260924 10:00:50.734180 21287 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0c6f777fd85462bb4da4d02642c3dc9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d0c6f777fd85462bb4da4d02642c3dc9. Latest consensus state: current_term: 1 leader_uuid: "d0c6f777fd85462bb4da4d02642c3dc9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0c6f777fd85462bb4da4d02642c3dc9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46405 } } }
I20260924 10:00:50.735095 21287 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0c6f777fd85462bb4da4d02642c3dc9 [sys.catalog]: This master's current role is: LEADER
I20260924 10:00:50.736557 21292 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:00:50.740928 21292 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:00:50.750977 21292 catalog_manager.cc:1347] Generated new cluster ID: 54471021d55245278ae1321a13abd33b
I20260924 10:00:50.751094 21292 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:00:50.762995 21292 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:00:50.764899 21292 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:00:50.784520 21292 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d0c6f777fd85462bb4da4d02642c3dc9: Generated new TSK 0
I20260924 10:00:50.785612 21292 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 10:00:50.797209 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu pid 21208
W20260924 10:00:50.798225 21270 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:36723
W20260924 10:00:50.798856 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu pid 21208: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 10:00:50.799005 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu with pid 21208
2026-09-24T10:00:50Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1798 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790243735634835-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:00:50 dist-test-slave-w8d2 krb5kdc[21306](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:00:50 dist-test-slave-w8d2 krb5kdc[21306](info): set up 2 sockets
Sep 24 10:00:50 dist-test-slave-w8d2 krb5kdc[21306](info): commencing operation
krb5kdc: starting...
W20260924 10:00:51.815342 388 mini_kdc.cc:121] Time spent starting KDC: real 0.985s user 0.000s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790243735634835-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790243735634835-388-0/krb5kdc/test-user.keytab.
Sep 24 10:00:51 dist-test-slave-w8d2 krb5kdc[21306](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244051, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:00:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:00:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790243735634835-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790243735634835-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 10:00:52.044384 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
/tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790243735634835-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790243735634835-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790243735634835-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790243735634835-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45257
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790243735634835-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45257
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790243735634835-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790243735634835-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790243735634835-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:00:52.359745 21319 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:00:52.360033 21319 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:00:52.360164 21319 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:00:52.367379 21319 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:00:52.367491 21319 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:00:52.367561 21319 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:00:52.367641 21319 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:00:52.367709 21319 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:00:52.377537 21319 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790243735634835-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790243735634835-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45257
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790243735634835-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45257
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790243735634835-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790243735634835-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:00:52.378830 21319 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:00:52.380155 21319 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:00:52.391523 21324 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:00:52.391597 21325 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:00:52.391597 21327 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:00:52.392560 21319 server_base.cc:1034] running on GCE node
I20260924 10:00:52.393362 21319 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:00:52.394759 21319 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:00:52.395951 21319 hybrid_clock.cc:648] HybridClock initialized: now 1790244052395905 us; error 39 us; skew 500 ppm
I20260924 10:00:52.396291 21319 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:00:52 dist-test-slave-w8d2 krb5kdc[21306](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244052, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:00:52.403891 21319 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 10:00:52.405452 21319 webserver.cc:460] Webserver started at http://127.0.0.1:39919/ using document root <none> and password file <none>
I20260924 10:00:52.406177 21319 fs_manager.cc:359] Metadata directory not provided
I20260924 10:00:52.406296 21319 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:00:52.406591 21319 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:00:52.409611 21319 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790243735634835-388-0/minicluster-data/master-0/data/instance:
uuid: "dd67bf4e8e164af09cc1628c9c73aa2d"
format_stamp: "Formatted at 2026-09-24 10:00:52 on dist-test-slave-w8d2"
server_key: "1f20f10f40de672b3801bb628ab3d1f3"
server_key_iv: "e64a9463ac90ab7fea61d75a34404407"
server_key_version: "encryptionkey@0"
I20260924 10:00:52.410367 21319 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790243735634835-388-0/minicluster-data/master-0/wal/instance:
uuid: "dd67bf4e8e164af09cc1628c9c73aa2d"
format_stamp: "Formatted at 2026-09-24 10:00:52 on dist-test-slave-w8d2"
server_key: "1f20f10f40de672b3801bb628ab3d1f3"
server_key_iv: "e64a9463ac90ab7fea61d75a34404407"
server_key_version: "encryptionkey@0"
I20260924 10:00:52.415642 21319 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260924 10:00:52.419018 21334 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:00:52.420399 21319 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260924 10:00:52.420562 21319 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790243735634835-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790243735634835-388-0/minicluster-data/master-0/wal
uuid: "dd67bf4e8e164af09cc1628c9c73aa2d"
format_stamp: "Formatted at 2026-09-24 10:00:52 on dist-test-slave-w8d2"
server_key: "1f20f10f40de672b3801bb628ab3d1f3"
server_key_iv: "e64a9463ac90ab7fea61d75a34404407"
server_key_version: "encryptionkey@0"
I20260924 10:00:52.420753 21319 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790243735634835-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790243735634835-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790243735634835-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:00:52.426457 21319 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:00:52.430693 21319 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:00:52.430966 21319 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:00:52.447753 21319 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45257
I20260924 10:00:52.447798 21385 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45257 every 8 connection(s)
I20260924 10:00:52.449106 21319 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790243735634835-388-0/minicluster-data/master-0/data/info.pb
I20260924 10:00:52.452796 21386 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:00:52.455888 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu as pid 21319
I20260924 10:00:52.456077 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790243735634835-388-0/minicluster-data/master-0/wal/instance
I20260924 10:00:52.456614 388 external_mini_cluster.cc:1428] Setting key 350adb256af44d01122b9148a099fbd9
I20260924 10:00:52.464181 21389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:52.458489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48339 (local address 127.0.0.1:45257)
0924 10:00:52.459266 (+ 777us) server_negotiation.cc:206] Beginning negotiation
0924 10:00:52.459277 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 10:00:52.459317 (+ 40us) server_negotiation.cc:407] Connection header received
0924 10:00:52.460956 (+ 1639us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:00:52.460978 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:00:52.461332 (+ 354us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:00:52.461705 (+ 373us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:00:52.462662 (+ 957us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:00:52.462668 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:00:52.462681 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:00:52.462695 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:00:52.462759 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:00:52.462783 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:00:52.462982 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0924 10:00:52.463101 (+ 119us) server_negotiation.cc:299] Negotiation successful
0924 10:00:52.463303 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":548,"thread_start_us":327,"threads_started":1}
I20260924 10:00:52.464582 21386 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd67bf4e8e164af09cc1628c9c73aa2d: Bootstrap starting.
I20260924 10:00:52.467779 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 10:00:52.468152 21386 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dd67bf4e8e164af09cc1628c9c73aa2d: Neither blocks nor log segments found. Creating new log.
I20260924 10:00:52.469096 21386 log.cc:824] T 00000000000000000000000000000000 P dd67bf4e8e164af09cc1628c9c73aa2d: Log is configured to *not* fsync() on all Append() calls
I20260924 10:00:52.471917 21386 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd67bf4e8e164af09cc1628c9c73aa2d: No bootstrap required, opened a new log
I20260924 10:00:52.474303 21389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:52.471900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60481 (local address 127.0.0.1:45257)
0924 10:00:52.472075 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0924 10:00:52.472082 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:00:52.472436 (+ 354us) server_negotiation.cc:407] Connection header received
0924 10:00:52.472623 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:00:52.472627 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:00:52.472674 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:00:52.472750 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:00:52.473665 (+ 915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:00:52.473669 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:00:52.473672 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:00:52.473689 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:00:52.473748 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:00:52.473762 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:00:52.473856 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0924 10:00:52.473959 (+ 103us) server_negotiation.cc:299] Negotiation successful
0924 10:00:52.474008 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 10:00:52.479637 21386 raft_consensus.cc:348] T 00000000000000000000000000000000 P dd67bf4e8e164af09cc1628c9c73aa2d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd67bf4e8e164af09cc1628c9c73aa2d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45257 } }
I20260924 10:00:52.480010 21386 raft_consensus.cc:374] T 00000000000000000000000000000000 P dd67bf4e8e164af09cc1628c9c73aa2d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:00:52.480111 21386 raft_consensus.cc:729] T 00000000000000000000000000000000 P dd67bf4e8e164af09cc1628c9c73aa2d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd67bf4e8e164af09cc1628c9c73aa2d, State: Initialized, Role: FOLLOWER
I20260924 10:00:52.480650 21386 consensus_queue.cc:260] T 00000000000000000000000000000000 P dd67bf4e8e164af09cc1628c9c73aa2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd67bf4e8e164af09cc1628c9c73aa2d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45257 } }
I20260924 10:00:52.480865 21386 raft_consensus.cc:388] T 00000000000000000000000000000000 P dd67bf4e8e164af09cc1628c9c73aa2d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:00:52.480983 21386 raft_consensus.cc:482] T 00000000000000000000000000000000 P dd67bf4e8e164af09cc1628c9c73aa2d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:00:52.481132 21386 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dd67bf4e8e164af09cc1628c9c73aa2d [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:00:52.483215 21386 raft_consensus.cc:504] T 00000000000000000000000000000000 P dd67bf4e8e164af09cc1628c9c73aa2d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd67bf4e8e164af09cc1628c9c73aa2d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45257 } }
I20260924 10:00:52.483762 21386 leader_election.cc:302] T 00000000000000000000000000000000 P dd67bf4e8e164af09cc1628c9c73aa2d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dd67bf4e8e164af09cc1628c9c73aa2d; no voters:
I20260924 10:00:52.484295 21386 leader_election.cc:288] T 00000000000000000000000000000000 P dd67bf4e8e164af09cc1628c9c73aa2d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:00:52.484417 21396 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dd67bf4e8e164af09cc1628c9c73aa2d [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:00:52.485092 21396 raft_consensus.cc:686] T 00000000000000000000000000000000 P dd67bf4e8e164af09cc1628c9c73aa2d [term 1 LEADER]: Becoming Leader. State: Replica: dd67bf4e8e164af09cc1628c9c73aa2d, State: Running, Role: LEADER
I20260924 10:00:52.485610 21396 consensus_queue.cc:237] T 00000000000000000000000000000000 P dd67bf4e8e164af09cc1628c9c73aa2d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd67bf4e8e164af09cc1628c9c73aa2d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45257 } }
I20260924 10:00:52.486339 21386 sys_catalog.cc:564] T 00000000000000000000000000000000 P dd67bf4e8e164af09cc1628c9c73aa2d [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:00:52.488368 21397 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd67bf4e8e164af09cc1628c9c73aa2d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dd67bf4e8e164af09cc1628c9c73aa2d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd67bf4e8e164af09cc1628c9c73aa2d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45257 } } }
I20260924 10:00:52.489058 21397 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd67bf4e8e164af09cc1628c9c73aa2d [sys.catalog]: This master's current role is: LEADER
I20260924 10:00:52.489010 21398 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd67bf4e8e164af09cc1628c9c73aa2d [sys.catalog]: SysCatalogTable state changed. Reason: New leader dd67bf4e8e164af09cc1628c9c73aa2d. Latest consensus state: current_term: 1 leader_uuid: "dd67bf4e8e164af09cc1628c9c73aa2d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd67bf4e8e164af09cc1628c9c73aa2d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45257 } } }
I20260924 10:00:52.489308 21398 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd67bf4e8e164af09cc1628c9c73aa2d [sys.catalog]: This master's current role is: LEADER
W20260924 10:00:52.497992 21412 catalog_manager.cc:1558] T 00000000000000000000000000000000 P dd67bf4e8e164af09cc1628c9c73aa2d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 10:00:52.498108 21412 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 10:00:52.498247 21404 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:00:52.499867 21404 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:00:52.508447 21404 catalog_manager.cc:1347] Generated new cluster ID: 7e4c7fa546e5432db9881ce981dba597
I20260924 10:00:52.508558 21404 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:00:52.518391 21404 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:00:52.518798 21404 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:00:52.539533 21404 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dd67bf4e8e164af09cc1628c9c73aa2d: Generated new TSK 0
I20260924 10:00:52.540405 21404 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 10:00:52.548189 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu pid 21319
W20260924 10:00:52.549245 21381 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:48339
W20260924 10:00:52.549913 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu pid 21319: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 10:00:52.550081 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu with pid 21319
2026-09-24T10:00:52Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1750 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790243735634835-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:00:52 dist-test-slave-w8d2 krb5kdc[21417](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:00:52 dist-test-slave-w8d2 krb5kdc[21417](info): set up 2 sockets
Sep 24 10:00:52 dist-test-slave-w8d2 krb5kdc[21417](info): commencing operation
krb5kdc: starting...
W20260924 10:00:53.559381 388 mini_kdc.cc:121] Time spent starting KDC: real 0.977s user 0.000s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790243735634835-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790243735634835-388-0/krb5kdc/test-user.keytab.
Sep 24 10:00:53 dist-test-slave-w8d2 krb5kdc[21417](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244053, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:00:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:00:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790243735634835-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790243735634835-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 10:00:53.781430 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
/tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790243735634835-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790243735634835-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790243735634835-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790243735634835-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37603
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790243735634835-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37603
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790243735634835-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790243735634835-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790243735634835-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:00:54.088589 21430 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:00:54.088897 21430 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:00:54.088999 21430 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:00:54.096221 21430 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:00:54.096334 21430 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:00:54.096379 21430 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:00:54.096441 21430 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:00:54.096483 21430 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:00:54.107291 21430 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790243735634835-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790243735634835-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37603
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790243735634835-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37603
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790243735634835-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790243735634835-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:00:54.108546 21430 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:00:54.109764 21430 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:00:54.120810 21435 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:00:54.120836 21436 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:00:54.120858 21438 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:00:54.121662 21430 server_base.cc:1034] running on GCE node
I20260924 10:00:54.122376 21430 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:00:54.123805 21430 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:00:54.124998 21430 hybrid_clock.cc:648] HybridClock initialized: now 1790244054124945 us; error 46 us; skew 500 ppm
I20260924 10:00:54.125325 21430 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:00:54 dist-test-slave-w8d2 krb5kdc[21417](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244054, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:00:54.128248 21430 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 10:00:54.129748 21430 webserver.cc:460] Webserver started at http://127.0.0.1:42335/ using document root <none> and password file <none>
I20260924 10:00:54.130409 21430 fs_manager.cc:359] Metadata directory not provided
I20260924 10:00:54.130524 21430 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:00:54.130836 21430 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:00:54.133858 21430 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790243735634835-388-0/minicluster-data/master-0/data/instance:
uuid: "99c5c297b11d4eac9f88d1f881da4fd3"
format_stamp: "Formatted at 2026-09-24 10:00:54 on dist-test-slave-w8d2"
server_key: "222f732ecb68f33e72497958502e6f83"
server_key_iv: "99b54c8ea2f5ef4cad26c781c5bf9bb7"
server_key_version: "encryptionkey@0"
I20260924 10:00:54.134647 21430 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790243735634835-388-0/minicluster-data/master-0/wal/instance:
uuid: "99c5c297b11d4eac9f88d1f881da4fd3"
format_stamp: "Formatted at 2026-09-24 10:00:54 on dist-test-slave-w8d2"
server_key: "222f732ecb68f33e72497958502e6f83"
server_key_iv: "99b54c8ea2f5ef4cad26c781c5bf9bb7"
server_key_version: "encryptionkey@0"
I20260924 10:00:54.139614 21430 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260924 10:00:54.142930 21445 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:00:54.144285 21430 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 10:00:54.144451 21430 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790243735634835-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790243735634835-388-0/minicluster-data/master-0/wal
uuid: "99c5c297b11d4eac9f88d1f881da4fd3"
format_stamp: "Formatted at 2026-09-24 10:00:54 on dist-test-slave-w8d2"
server_key: "222f732ecb68f33e72497958502e6f83"
server_key_iv: "99b54c8ea2f5ef4cad26c781c5bf9bb7"
server_key_version: "encryptionkey@0"
I20260924 10:00:54.144640 21430 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790243735634835-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790243735634835-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790243735634835-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:00:54.160893 21430 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:00:54.164775 21430 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:00:54.165047 21430 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:00:54.182135 21430 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37603
I20260924 10:00:54.182200 21496 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37603 every 8 connection(s)
I20260924 10:00:54.183465 21430 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790243735634835-388-0/minicluster-data/master-0/data/info.pb
I20260924 10:00:54.187094 21497 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:00:54.192476 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu as pid 21430
I20260924 10:00:54.192651 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790243735634835-388-0/minicluster-data/master-0/wal/instance
I20260924 10:00:54.193176 388 external_mini_cluster.cc:1428] Setting key 08055904e142d914586353727a0445a9
I20260924 10:00:54.197796 21497 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 99c5c297b11d4eac9f88d1f881da4fd3: Bootstrap starting.
I20260924 10:00:54.200690 21497 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 99c5c297b11d4eac9f88d1f881da4fd3: Neither blocks nor log segments found. Creating new log.
I20260924 10:00:54.201630 21497 log.cc:824] T 00000000000000000000000000000000 P 99c5c297b11d4eac9f88d1f881da4fd3: Log is configured to *not* fsync() on all Append() calls
I20260924 10:00:54.204275 21497 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 99c5c297b11d4eac9f88d1f881da4fd3: No bootstrap required, opened a new log
Sep 24 10:00:54 dist-test-slave-w8d2 krb5kdc[21417](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244053, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 10:00:54.213122 21497 raft_consensus.cc:348] T 00000000000000000000000000000000 P 99c5c297b11d4eac9f88d1f881da4fd3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99c5c297b11d4eac9f88d1f881da4fd3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37603 } }
I20260924 10:00:54.213565 21497 raft_consensus.cc:374] T 00000000000000000000000000000000 P 99c5c297b11d4eac9f88d1f881da4fd3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:00:54.213670 21497 raft_consensus.cc:729] T 00000000000000000000000000000000 P 99c5c297b11d4eac9f88d1f881da4fd3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99c5c297b11d4eac9f88d1f881da4fd3, State: Initialized, Role: FOLLOWER
I20260924 10:00:54.214274 21497 consensus_queue.cc:260] T 00000000000000000000000000000000 P 99c5c297b11d4eac9f88d1f881da4fd3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99c5c297b11d4eac9f88d1f881da4fd3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37603 } }
I20260924 10:00:54.214473 21500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:54.194971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.162:46589 (local address 127.0.0.1:37603)
0924 10:00:54.195755 (+ 784us) server_negotiation.cc:206] Beginning negotiation
0924 10:00:54.195766 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 10:00:54.195808 (+ 42us) server_negotiation.cc:407] Connection header received
0924 10:00:54.197447 (+ 1639us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:00:54.197468 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:00:54.197844 (+ 376us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:00:54.198206 (+ 362us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:00:54.199586 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:00:54.200832 (+ 1246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:00:54.202050 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:00:54.202730 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:00:54.202882 (+ 152us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:00:54.207140 (+ 4258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:00:54.207148 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:00:54.207163 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:00:54.207202 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:00:54.210229 (+ 3027us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:00:54.211158 (+ 929us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:00:54.211164 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:00:54.211172 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:00:54.211371 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:00:54.211821 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:00:54.211824 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:00:54.211826 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:00:54.212375 (+ 549us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:00:54.212599 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0924 10:00:54.213043 (+ 444us) server_negotiation.cc:299] Negotiation successful
0924 10:00:54.213330 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":567,"thread_start_us":327,"threads_started":1}
I20260924 10:00:54.214542 21497 raft_consensus.cc:388] T 00000000000000000000000000000000 P 99c5c297b11d4eac9f88d1f881da4fd3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:00:54.214689 21497 raft_consensus.cc:482] T 00000000000000000000000000000000 P 99c5c297b11d4eac9f88d1f881da4fd3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:00:54.214869 21497 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 99c5c297b11d4eac9f88d1f881da4fd3 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:00:54.217727 21497 raft_consensus.cc:504] T 00000000000000000000000000000000 P 99c5c297b11d4eac9f88d1f881da4fd3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99c5c297b11d4eac9f88d1f881da4fd3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37603 } }
I20260924 10:00:54.218304 21497 leader_election.cc:302] T 00000000000000000000000000000000 P 99c5c297b11d4eac9f88d1f881da4fd3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 99c5c297b11d4eac9f88d1f881da4fd3; no voters:
I20260924 10:00:54.218518 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 10:00:54.218904 21497 leader_election.cc:288] T 00000000000000000000000000000000 P 99c5c297b11d4eac9f88d1f881da4fd3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:00:54.219044 21502 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 99c5c297b11d4eac9f88d1f881da4fd3 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:00:54.220819 21497 sys_catalog.cc:564] T 00000000000000000000000000000000 P 99c5c297b11d4eac9f88d1f881da4fd3 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:00:54.228544 21502 raft_consensus.cc:686] T 00000000000000000000000000000000 P 99c5c297b11d4eac9f88d1f881da4fd3 [term 1 LEADER]: Becoming Leader. State: Replica: 99c5c297b11d4eac9f88d1f881da4fd3, State: Running, Role: LEADER
I20260924 10:00:54.229439 21502 consensus_queue.cc:237] T 00000000000000000000000000000000 P 99c5c297b11d4eac9f88d1f881da4fd3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99c5c297b11d4eac9f88d1f881da4fd3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37603 } }
I20260924 10:00:54.232714 21503 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99c5c297b11d4eac9f88d1f881da4fd3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "99c5c297b11d4eac9f88d1f881da4fd3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99c5c297b11d4eac9f88d1f881da4fd3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37603 } } }
I20260924 10:00:54.233129 21503 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99c5c297b11d4eac9f88d1f881da4fd3 [sys.catalog]: This master's current role is: LEADER
I20260924 10:00:54.233075 21504 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99c5c297b11d4eac9f88d1f881da4fd3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 99c5c297b11d4eac9f88d1f881da4fd3. Latest consensus state: current_term: 1 leader_uuid: "99c5c297b11d4eac9f88d1f881da4fd3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99c5c297b11d4eac9f88d1f881da4fd3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37603 } } }
I20260924 10:00:54.233384 21504 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99c5c297b11d4eac9f88d1f881da4fd3 [sys.catalog]: This master's current role is: LEADER
I20260924 10:00:54.239866 21522 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:00:54.244913 21522 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:00:54.252418 21500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:54.231778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.162:51901 (local address 127.0.0.1:37603)
0924 10:00:54.231937 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0924 10:00:54.231943 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:00:54.235474 (+ 3531us) server_negotiation.cc:407] Connection header received
0924 10:00:54.235684 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:00:54.235690 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:00:54.235781 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:00:54.235899 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:00:54.237902 (+ 2003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:00:54.239136 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:00:54.240714 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:00:54.241330 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:00:54.241471 (+ 141us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:00:54.244440 (+ 2969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:00:54.244447 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:00:54.244456 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:00:54.244498 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:00:54.248388 (+ 3890us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:00:54.249564 (+ 1176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:00:54.249569 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:00:54.249572 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:00:54.249725 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:00:54.250225 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:00:54.250229 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:00:54.250232 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:00:54.250616 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:00:54.250858 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0924 10:00:54.251282 (+ 424us) server_negotiation.cc:299] Negotiation successful
0924 10:00:54.252036 (+ 754us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260924 10:00:54.254890 21522 catalog_manager.cc:1347] Generated new cluster ID: 9a3a21a8db024678b59ef9c3b2e86ccb
I20260924 10:00:54.254990 21522 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:00:54.272671 21522 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:00:54.273952 21522 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:00:54.279965 21522 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 99c5c297b11d4eac9f88d1f881da4fd3: Generated new TSK 0
I20260924 10:00:54.281097 21522 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 10:00:54.290848 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu pid 21430
I20260924 10:00:54.292872 21492 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:00:54.383466 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu with pid 21430
W20260924 10:00:54.397483 21415 connection.cc:535] client connection to 127.0.0.1:37603 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37603): Connection reset by peer (error 104)
2026-09-24T10:00:54Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1834 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790243735634835-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:00:54 dist-test-slave-w8d2 krb5kdc[21529](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:00:54 dist-test-slave-w8d2 krb5kdc[21529](info): set up 2 sockets
Sep 24 10:00:54 dist-test-slave-w8d2 krb5kdc[21529](info): commencing operation
krb5kdc: starting...
W20260924 10:00:55.399421 388 mini_kdc.cc:121] Time spent starting KDC: real 0.983s user 0.003s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790243735634835-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790243735634835-388-0/krb5kdc/test-user.keytab.
Sep 24 10:00:55 dist-test-slave-w8d2 krb5kdc[21529](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244055, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:00:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:00:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790243735634835-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790243735634835-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 10:00:55.626504 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
/tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790243735634835-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790243735634835-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790243735634835-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790243735634835-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38543
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790243735634835-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38543
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790243735634835-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790243735634835-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790243735634835-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:00:55.933535 21542 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:00:55.933813 21542 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:00:55.933947 21542 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:00:55.941133 21542 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:00:55.941236 21542 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:00:55.941308 21542 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:00:55.941393 21542 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:00:55.941447 21542 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:00:55.951164 21542 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790243735634835-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790243735634835-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38543
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790243735634835-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38543
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790243735634835-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790243735634835-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:00:55.952463 21542 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:00:55.953728 21542 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:00:55.965139 21547 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:00:55.965114 21548 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:00:55.965075 21550 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:00:55.965647 21542 server_base.cc:1034] running on GCE node
I20260924 10:00:55.966558 21542 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:00:55.968871 21542 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:00:55.970070 21542 hybrid_clock.cc:648] HybridClock initialized: now 1790244055970013 us; error 51 us; skew 500 ppm
I20260924 10:00:55.970396 21542 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:00:55 dist-test-slave-w8d2 krb5kdc[21529](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244055, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:00:55.973327 21542 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 10:00:55.974646 21542 webserver.cc:460] Webserver started at http://127.0.0.1:33251/ using document root <none> and password file <none>
I20260924 10:00:55.975322 21542 fs_manager.cc:359] Metadata directory not provided
I20260924 10:00:55.975436 21542 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:00:55.975706 21542 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:00:55.978485 21542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790243735634835-388-0/minicluster-data/master-0/data/instance:
uuid: "15c06bbe6bde4af0b97a64c7982d4804"
format_stamp: "Formatted at 2026-09-24 10:00:55 on dist-test-slave-w8d2"
server_key: "615b87b15c091b28a945e4ad7324a333"
server_key_iv: "cd875745a30a9d338f3cedf91c5fa083"
server_key_version: "encryptionkey@0"
I20260924 10:00:55.979175 21542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790243735634835-388-0/minicluster-data/master-0/wal/instance:
uuid: "15c06bbe6bde4af0b97a64c7982d4804"
format_stamp: "Formatted at 2026-09-24 10:00:55 on dist-test-slave-w8d2"
server_key: "615b87b15c091b28a945e4ad7324a333"
server_key_iv: "cd875745a30a9d338f3cedf91c5fa083"
server_key_version: "encryptionkey@0"
I20260924 10:00:55.983911 21542 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 10:00:55.987075 21557 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:00:55.988338 21542 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260924 10:00:55.988523 21542 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790243735634835-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790243735634835-388-0/minicluster-data/master-0/wal
uuid: "15c06bbe6bde4af0b97a64c7982d4804"
format_stamp: "Formatted at 2026-09-24 10:00:55 on dist-test-slave-w8d2"
server_key: "615b87b15c091b28a945e4ad7324a333"
server_key_iv: "cd875745a30a9d338f3cedf91c5fa083"
server_key_version: "encryptionkey@0"
I20260924 10:00:55.988691 21542 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790243735634835-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790243735634835-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790243735634835-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:00:56.015026 21542 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:00:56.018659 21542 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:00:56.018906 21542 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:00:56.034824 21542 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38543
I20260924 10:00:56.034827 21608 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38543 every 8 connection(s)
I20260924 10:00:56.036191 21542 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790243735634835-388-0/minicluster-data/master-0/data/info.pb
I20260924 10:00:56.039728 21609 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:00:56.045951 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu as pid 21542
I20260924 10:00:56.046110 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790243735634835-388-0/minicluster-data/master-0/wal/instance
I20260924 10:00:56.046485 388 external_mini_cluster.cc:1428] Setting key 4b71ad9b76233102836fce87590e8919
I20260924 10:00:56.049750 21609 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 15c06bbe6bde4af0b97a64c7982d4804: Bootstrap starting.
I20260924 10:00:56.053751 21609 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 15c06bbe6bde4af0b97a64c7982d4804: Neither blocks nor log segments found. Creating new log.
I20260924 10:00:56.054960 21609 log.cc:824] T 00000000000000000000000000000000 P 15c06bbe6bde4af0b97a64c7982d4804: Log is configured to *not* fsync() on all Append() calls
I20260924 10:00:56.057981 21609 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 15c06bbe6bde4af0b97a64c7982d4804: No bootstrap required, opened a new log
I20260924 10:00:56.061275 21612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:56.048195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.162:44029 (local address 127.0.0.1:38543)
0924 10:00:56.048909 (+ 714us) server_negotiation.cc:206] Beginning negotiation
0924 10:00:56.048923 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 10:00:56.048968 (+ 45us) server_negotiation.cc:407] Connection header received
0924 10:00:56.051414 (+ 2446us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:00:56.051443 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:00:56.052014 (+ 571us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:00:56.052441 (+ 427us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:00:56.052937 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:00:56.054813 (+ 1876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:00:56.056469 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:00:56.057447 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:00:56.057613 (+ 166us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:00:56.058633 (+ 1020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:00:56.058644 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:00:56.058662 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:00:56.059366 (+ 704us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:00:56.059454 (+ 88us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:00:56.059487 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:00:56.059659 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0924 10:00:56.059850 (+ 191us) server_negotiation.cc:299] Negotiation successful
0924 10:00:56.060035 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":489,"thread_start_us":296,"threads_started":1}
I20260924 10:00:56.064791 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 10:00:56.066939 21609 raft_consensus.cc:348] T 00000000000000000000000000000000 P 15c06bbe6bde4af0b97a64c7982d4804 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15c06bbe6bde4af0b97a64c7982d4804" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38543 } }
I20260924 10:00:56.067384 21609 raft_consensus.cc:374] T 00000000000000000000000000000000 P 15c06bbe6bde4af0b97a64c7982d4804 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:00:56.067492 21609 raft_consensus.cc:729] T 00000000000000000000000000000000 P 15c06bbe6bde4af0b97a64c7982d4804 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15c06bbe6bde4af0b97a64c7982d4804, State: Initialized, Role: FOLLOWER
I20260924 10:00:56.068128 21609 consensus_queue.cc:260] T 00000000000000000000000000000000 P 15c06bbe6bde4af0b97a64c7982d4804 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15c06bbe6bde4af0b97a64c7982d4804" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38543 } }
I20260924 10:00:56.068414 21609 raft_consensus.cc:388] T 00000000000000000000000000000000 P 15c06bbe6bde4af0b97a64c7982d4804 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:00:56.068571 21609 raft_consensus.cc:482] T 00000000000000000000000000000000 P 15c06bbe6bde4af0b97a64c7982d4804 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:00:56.068739 21609 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 15c06bbe6bde4af0b97a64c7982d4804 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:00:56.071570 21609 raft_consensus.cc:504] T 00000000000000000000000000000000 P 15c06bbe6bde4af0b97a64c7982d4804 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15c06bbe6bde4af0b97a64c7982d4804" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38543 } }
I20260924 10:00:56.072036 21609 leader_election.cc:302] T 00000000000000000000000000000000 P 15c06bbe6bde4af0b97a64c7982d4804 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 15c06bbe6bde4af0b97a64c7982d4804; no voters:
I20260924 10:00:56.072665 21609 leader_election.cc:288] T 00000000000000000000000000000000 P 15c06bbe6bde4af0b97a64c7982d4804 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:00:56.072731 21619 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 15c06bbe6bde4af0b97a64c7982d4804 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:00:56.074456 21609 sys_catalog.cc:564] T 00000000000000000000000000000000 P 15c06bbe6bde4af0b97a64c7982d4804 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:00:56.074795 21619 raft_consensus.cc:686] T 00000000000000000000000000000000 P 15c06bbe6bde4af0b97a64c7982d4804 [term 1 LEADER]: Becoming Leader. State: Replica: 15c06bbe6bde4af0b97a64c7982d4804, State: Running, Role: LEADER
I20260924 10:00:56.075445 21619 consensus_queue.cc:237] T 00000000000000000000000000000000 P 15c06bbe6bde4af0b97a64c7982d4804 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15c06bbe6bde4af0b97a64c7982d4804" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38543 } }
I20260924 10:00:56.077054 21612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:56.069604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.162:45699 (local address 127.0.0.1:38543)
0924 10:00:56.069780 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0924 10:00:56.069786 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:00:56.070270 (+ 484us) server_negotiation.cc:407] Connection header received
0924 10:00:56.070417 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:00:56.070421 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:00:56.070480 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:00:56.070552 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:00:56.071102 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:00:56.071905 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:00:56.073780 (+ 1875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:00:56.074517 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:00:56.074651 (+ 134us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:00:56.076129 (+ 1478us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:00:56.076135 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:00:56.076139 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:00:56.076159 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:00:56.076233 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:00:56.076255 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:00:56.076349 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0924 10:00:56.076663 (+ 314us) server_negotiation.cc:299] Negotiation successful
0924 10:00:56.076714 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 10:00:56.078248 21620 sys_catalog.cc:455] T 00000000000000000000000000000000 P 15c06bbe6bde4af0b97a64c7982d4804 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "15c06bbe6bde4af0b97a64c7982d4804" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15c06bbe6bde4af0b97a64c7982d4804" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38543 } } }
I20260924 10:00:56.078619 21620 sys_catalog.cc:458] T 00000000000000000000000000000000 P 15c06bbe6bde4af0b97a64c7982d4804 [sys.catalog]: This master's current role is: LEADER
I20260924 10:00:56.079272 21621 sys_catalog.cc:455] T 00000000000000000000000000000000 P 15c06bbe6bde4af0b97a64c7982d4804 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 15c06bbe6bde4af0b97a64c7982d4804. Latest consensus state: current_term: 1 leader_uuid: "15c06bbe6bde4af0b97a64c7982d4804" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15c06bbe6bde4af0b97a64c7982d4804" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38543 } } }
I20260924 10:00:56.079608 21621 sys_catalog.cc:458] T 00000000000000000000000000000000 P 15c06bbe6bde4af0b97a64c7982d4804 [sys.catalog]: This master's current role is: LEADER
I20260924 10:00:56.083102 21628 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:00:56.088145 21628 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:00:56.095559 21628 catalog_manager.cc:1347] Generated new cluster ID: f82e403e22c94d1ea2983399d236e443
I20260924 10:00:56.095654 21628 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:00:56.132131 21628 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:00:56.134100 21628 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:00:56.147625 21628 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 15c06bbe6bde4af0b97a64c7982d4804: Generated new TSK 0
I20260924 10:00:56.148420 21628 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 10:00:56.162693 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu pid 21542
W20260924 10:00:56.164408 21604 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.162:44029
W20260924 10:00:56.165158 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu pid 21542: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 10:00:56.165277 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu with pid 21542
W20260924 10:00:56.178821 21527 connection.cc:535] client connection to 127.0.0.1:38543 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38543): Connection reset by peer (error 104)
2026-09-24T10:00:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1781 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790243735634835-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:00:56 dist-test-slave-w8d2 krb5kdc[21640](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:00:56 dist-test-slave-w8d2 krb5kdc[21640](info): set up 2 sockets
Sep 24 10:00:56 dist-test-slave-w8d2 krb5kdc[21640](info): commencing operation
krb5kdc: starting...
W20260924 10:00:57.179376 388 mini_kdc.cc:121] Time spent starting KDC: real 0.982s user 0.000s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790243735634835-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790243735634835-388-0/krb5kdc/test-user.keytab.
Sep 24 10:00:57 dist-test-slave-w8d2 krb5kdc[21640](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244057, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:00:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:00:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790243735634835-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790243735634835-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 10:00:57.402251 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
/tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790243735634835-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790243735634835-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790243735634835-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790243735634835-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45133
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790243735634835-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45133
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790243735634835-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790243735634835-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790243735634835-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:00:57.708179 21653 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:00:57.708452 21653 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:00:57.708588 21653 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:00:57.715541 21653 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:00:57.715660 21653 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:00:57.715734 21653 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:00:57.715811 21653 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:00:57.715865 21653 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:00:57.725350 21653 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790243735634835-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790243735634835-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45133
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790243735634835-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45133
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790243735634835-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790243735634835-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:00:57.726511 21653 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:00:57.727782 21653 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:00:57.738953 21659 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:00:57.739033 21658 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:00:57.739063 21661 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:00:57.739606 21653 server_base.cc:1034] running on GCE node
I20260924 10:00:57.740327 21653 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:00:57.741716 21653 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:00:57.742893 21653 hybrid_clock.cc:648] HybridClock initialized: now 1790244057742840 us; error 52 us; skew 500 ppm
I20260924 10:00:57.743196 21653 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:00:57 dist-test-slave-w8d2 krb5kdc[21640](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244057, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:00:57.751492 21653 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 10:00:57.752825 21653 webserver.cc:460] Webserver started at http://127.0.0.1:46187/ using document root <none> and password file <none>
I20260924 10:00:57.753437 21653 fs_manager.cc:359] Metadata directory not provided
I20260924 10:00:57.753553 21653 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:00:57.753821 21653 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:00:57.756635 21653 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790243735634835-388-0/minicluster-data/master-0/data/instance:
uuid: "800d282104474cc2838e9fa7a8d1a2a6"
format_stamp: "Formatted at 2026-09-24 10:00:57 on dist-test-slave-w8d2"
server_key: "3dd0fcb32a23018814d7f29de75cdea2"
server_key_iv: "21e12145f848b468d4eefe643e3f98ac"
server_key_version: "encryptionkey@0"
I20260924 10:00:57.757309 21653 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790243735634835-388-0/minicluster-data/master-0/wal/instance:
uuid: "800d282104474cc2838e9fa7a8d1a2a6"
format_stamp: "Formatted at 2026-09-24 10:00:57 on dist-test-slave-w8d2"
server_key: "3dd0fcb32a23018814d7f29de75cdea2"
server_key_iv: "21e12145f848b468d4eefe643e3f98ac"
server_key_version: "encryptionkey@0"
I20260924 10:00:57.761972 21653 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 10:00:57.765015 21668 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:00:57.766147 21653 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 10:00:57.766291 21653 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790243735634835-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790243735634835-388-0/minicluster-data/master-0/wal
uuid: "800d282104474cc2838e9fa7a8d1a2a6"
format_stamp: "Formatted at 2026-09-24 10:00:57 on dist-test-slave-w8d2"
server_key: "3dd0fcb32a23018814d7f29de75cdea2"
server_key_iv: "21e12145f848b468d4eefe643e3f98ac"
server_key_version: "encryptionkey@0"
I20260924 10:00:57.766463 21653 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790243735634835-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790243735634835-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790243735634835-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:00:57.771571 21653 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:00:57.775271 21653 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:00:57.775523 21653 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:00:57.791896 21653 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45133
I20260924 10:00:57.791904 21719 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45133 every 8 connection(s)
I20260924 10:00:57.793143 21653 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790243735634835-388-0/minicluster-data/master-0/data/info.pb
I20260924 10:00:57.796547 21720 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:00:57.802129 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu as pid 21653
I20260924 10:00:57.802279 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790243735634835-388-0/minicluster-data/master-0/wal/instance
I20260924 10:00:57.802672 388 external_mini_cluster.cc:1428] Setting key 17fad69900092ba23efdd8b7cd76f488
I20260924 10:00:57.806620 21720 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 800d282104474cc2838e9fa7a8d1a2a6: Bootstrap starting.
I20260924 10:00:57.809953 21720 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 800d282104474cc2838e9fa7a8d1a2a6: Neither blocks nor log segments found. Creating new log.
I20260924 10:00:57.809971 21723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:57.804408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.162:46161 (local address 127.0.0.1:45133)
0924 10:00:57.805183 (+ 775us) server_negotiation.cc:206] Beginning negotiation
0924 10:00:57.805195 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 10:00:57.805233 (+ 38us) server_negotiation.cc:407] Connection header received
0924 10:00:57.806843 (+ 1610us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:00:57.806866 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:00:57.807194 (+ 328us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:00:57.807563 (+ 369us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:00:57.808541 (+ 978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:00:57.808547 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:00:57.808560 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:00:57.808576 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:00:57.808634 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:00:57.808653 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:00:57.808813 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0924 10:00:57.808928 (+ 115us) server_negotiation.cc:299] Negotiation successful
0924 10:00:57.809100 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":565,"thread_start_us":351,"threads_started":1}
I20260924 10:00:57.810932 21720 log.cc:824] T 00000000000000000000000000000000 P 800d282104474cc2838e9fa7a8d1a2a6: Log is configured to *not* fsync() on all Append() calls
I20260924 10:00:57.813360 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 10:00:57.813724 21720 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 800d282104474cc2838e9fa7a8d1a2a6: No bootstrap required, opened a new log
I20260924 10:00:57.819614 21723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:57.817392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.162:43689 (local address 127.0.0.1:45133)
0924 10:00:57.817584 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0924 10:00:57.817590 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:00:57.817986 (+ 396us) server_negotiation.cc:407] Connection header received
0924 10:00:57.818108 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:00:57.818113 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:00:57.818167 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:00:57.818265 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:00:57.819001 (+ 736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:00:57.819006 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:00:57.819011 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:00:57.819028 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:00:57.819075 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:00:57.819093 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:00:57.819216 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0924 10:00:57.819283 (+ 67us) server_negotiation.cc:299] Negotiation successful
0924 10:00:57.819333 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 10:00:57.821326 21720 raft_consensus.cc:348] T 00000000000000000000000000000000 P 800d282104474cc2838e9fa7a8d1a2a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "800d282104474cc2838e9fa7a8d1a2a6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45133 } }
I20260924 10:00:57.821638 21720 raft_consensus.cc:374] T 00000000000000000000000000000000 P 800d282104474cc2838e9fa7a8d1a2a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:00:57.821704 21720 raft_consensus.cc:729] T 00000000000000000000000000000000 P 800d282104474cc2838e9fa7a8d1a2a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 800d282104474cc2838e9fa7a8d1a2a6, State: Initialized, Role: FOLLOWER
I20260924 10:00:57.822186 21720 consensus_queue.cc:260] T 00000000000000000000000000000000 P 800d282104474cc2838e9fa7a8d1a2a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "800d282104474cc2838e9fa7a8d1a2a6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45133 } }
I20260924 10:00:57.822386 21720 raft_consensus.cc:388] T 00000000000000000000000000000000 P 800d282104474cc2838e9fa7a8d1a2a6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:00:57.822476 21720 raft_consensus.cc:482] T 00000000000000000000000000000000 P 800d282104474cc2838e9fa7a8d1a2a6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:00:57.822598 21720 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 800d282104474cc2838e9fa7a8d1a2a6 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:00:57.824610 21720 raft_consensus.cc:504] T 00000000000000000000000000000000 P 800d282104474cc2838e9fa7a8d1a2a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "800d282104474cc2838e9fa7a8d1a2a6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45133 } }
I20260924 10:00:57.825013 21720 leader_election.cc:302] T 00000000000000000000000000000000 P 800d282104474cc2838e9fa7a8d1a2a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 800d282104474cc2838e9fa7a8d1a2a6; no voters:
I20260924 10:00:57.825484 21720 leader_election.cc:288] T 00000000000000000000000000000000 P 800d282104474cc2838e9fa7a8d1a2a6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:00:57.825599 21730 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 800d282104474cc2838e9fa7a8d1a2a6 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:00:57.826819 21730 raft_consensus.cc:686] T 00000000000000000000000000000000 P 800d282104474cc2838e9fa7a8d1a2a6 [term 1 LEADER]: Becoming Leader. State: Replica: 800d282104474cc2838e9fa7a8d1a2a6, State: Running, Role: LEADER
I20260924 10:00:57.827394 21730 consensus_queue.cc:237] T 00000000000000000000000000000000 P 800d282104474cc2838e9fa7a8d1a2a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "800d282104474cc2838e9fa7a8d1a2a6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45133 } }
I20260924 10:00:57.827623 21720 sys_catalog.cc:564] T 00000000000000000000000000000000 P 800d282104474cc2838e9fa7a8d1a2a6 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:00:57.829802 21731 sys_catalog.cc:455] T 00000000000000000000000000000000 P 800d282104474cc2838e9fa7a8d1a2a6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "800d282104474cc2838e9fa7a8d1a2a6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "800d282104474cc2838e9fa7a8d1a2a6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45133 } } }
I20260924 10:00:57.830289 21731 sys_catalog.cc:458] T 00000000000000000000000000000000 P 800d282104474cc2838e9fa7a8d1a2a6 [sys.catalog]: This master's current role is: LEADER
I20260924 10:00:57.830317 21732 sys_catalog.cc:455] T 00000000000000000000000000000000 P 800d282104474cc2838e9fa7a8d1a2a6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 800d282104474cc2838e9fa7a8d1a2a6. Latest consensus state: current_term: 1 leader_uuid: "800d282104474cc2838e9fa7a8d1a2a6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "800d282104474cc2838e9fa7a8d1a2a6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45133 } } }
I20260924 10:00:57.830580 21732 sys_catalog.cc:458] T 00000000000000000000000000000000 P 800d282104474cc2838e9fa7a8d1a2a6 [sys.catalog]: This master's current role is: LEADER
I20260924 10:00:57.831826 21739 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:00:57.836611 21739 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:00:57.844543 21739 catalog_manager.cc:1347] Generated new cluster ID: 514bfee085ef4594a78d544aacba8ceb
I20260924 10:00:57.844640 21739 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:00:57.862198 21739 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:00:57.862566 21739 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:00:57.874205 21739 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 800d282104474cc2838e9fa7a8d1a2a6: Generated new TSK 0
I20260924 10:00:57.874883 21739 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 10:00:57.884456 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu pid 21653
W20260924 10:00:57.885444 21715 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.162:46161
W20260924 10:00:57.886107 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu pid 21653: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 10:00:57.886233 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu with pid 21653
2026-09-24T10:00:57Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1720 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790243735634835-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:00:57 dist-test-slave-w8d2 krb5kdc[21751](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:00:57 dist-test-slave-w8d2 krb5kdc[21751](info): set up 2 sockets
Sep 24 10:00:57 dist-test-slave-w8d2 krb5kdc[21751](info): commencing operation
krb5kdc: starting...
W20260924 10:00:58.887370 388 mini_kdc.cc:121] Time spent starting KDC: real 0.970s user 0.000s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790243735634835-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790243735634835-388-0/krb5kdc/test-user.keytab.
Sep 24 10:00:59 dist-test-slave-w8d2 krb5kdc[21751](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244059, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:00:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:00:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790243735634835-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790243735634835-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 10:00:59.103821 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
/tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790243735634835-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790243735634835-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790243735634835-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790243735634835-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41091
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790243735634835-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45443
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41091
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790243735634835-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790243735634835-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790243735634835-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:00:59.409502 21764 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:00:59.409786 21764 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:00:59.409931 21764 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:00:59.417351 21764 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:00:59.417464 21764 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:00:59.417546 21764 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:00:59.417625 21764 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:00:59.417682 21764 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:00:59.427083 21764 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45443
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790243735634835-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790243735634835-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41091
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790243735634835-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41091
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790243735634835-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790243735634835-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:00:59.428298 21764 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:00:59.429523 21764 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:00:59.440138 21769 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:00:59.440253 21770 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:00:59.440382 21772 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:00:59.441370 21764 server_base.cc:1034] running on GCE node
I20260924 10:00:59.442095 21764 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:00:59.443399 21764 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:00:59.444590 21764 hybrid_clock.cc:648] HybridClock initialized: now 1790244059444557 us; error 26 us; skew 500 ppm
I20260924 10:00:59.444885 21764 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:00:59 dist-test-slave-w8d2 krb5kdc[21751](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244059, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:00:59.447708 21764 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 10:00:59.448998 21764 webserver.cc:460] Webserver started at http://127.0.0.1:35955/ using document root <none> and password file <none>
I20260924 10:00:59.449584 21764 fs_manager.cc:359] Metadata directory not provided
I20260924 10:00:59.449680 21764 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:00:59.449924 21764 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:00:59.452688 21764 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790243735634835-388-0/minicluster-data/master-0/data/instance:
uuid: "40023ae58d5e494381581fc98d1b056b"
format_stamp: "Formatted at 2026-09-24 10:00:59 on dist-test-slave-w8d2"
server_key: "e2413e7522d481da06d5b8e047c49ec4"
server_key_iv: "9e5c4f3496d647083f52b952f49ee5f8"
server_key_version: "encryptionkey@0"
I20260924 10:00:59.453425 21764 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790243735634835-388-0/minicluster-data/master-0/wal/instance:
uuid: "40023ae58d5e494381581fc98d1b056b"
format_stamp: "Formatted at 2026-09-24 10:00:59 on dist-test-slave-w8d2"
server_key: "e2413e7522d481da06d5b8e047c49ec4"
server_key_iv: "9e5c4f3496d647083f52b952f49ee5f8"
server_key_version: "encryptionkey@0"
I20260924 10:00:59.458145 21764 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.005s
I20260924 10:00:59.461335 21779 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:00:59.462603 21764 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260924 10:00:59.462730 21764 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790243735634835-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790243735634835-388-0/minicluster-data/master-0/wal
uuid: "40023ae58d5e494381581fc98d1b056b"
format_stamp: "Formatted at 2026-09-24 10:00:59 on dist-test-slave-w8d2"
server_key: "e2413e7522d481da06d5b8e047c49ec4"
server_key_iv: "9e5c4f3496d647083f52b952f49ee5f8"
server_key_version: "encryptionkey@0"
I20260924 10:00:59.462877 21764 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790243735634835-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790243735634835-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790243735634835-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:00:59.487841 21764 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:00:59.491814 21764 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:00:59.492034 21764 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:00:59.508165 21764 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41091
I20260924 10:00:59.508184 21830 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41091 every 8 connection(s)
I20260924 10:00:59.509428 21764 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790243735634835-388-0/minicluster-data/master-0/data/info.pb
I20260924 10:00:59.512876 21831 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:00:59.514150 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu as pid 21764
I20260924 10:00:59.514295 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790243735634835-388-0/minicluster-data/master-0/wal/instance
I20260924 10:00:59.514707 388 external_mini_cluster.cc:1428] Setting key c86b145f08feabf02cff92ca6deeb4ee
I20260924 10:00:59.525768 21831 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 40023ae58d5e494381581fc98d1b056b: Bootstrap starting.
Sep 24 10:00:59 dist-test-slave-w8d2 krb5kdc[21751](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244059, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 10:00:59.529351 21831 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 40023ae58d5e494381581fc98d1b056b: Neither blocks nor log segments found. Creating new log.
I20260924 10:00:59.530308 21831 log.cc:824] T 00000000000000000000000000000000 P 40023ae58d5e494381581fc98d1b056b: Log is configured to *not* fsync() on all Append() calls
I20260924 10:00:59.533056 21831 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 40023ae58d5e494381581fc98d1b056b: No bootstrap required, opened a new log
I20260924 10:00:59.536718 21834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:59.516389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.162:41919 (local address 127.0.0.1:41091)
0924 10:00:59.517112 (+ 723us) server_negotiation.cc:206] Beginning negotiation
0924 10:00:59.517124 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 10:00:59.517165 (+ 41us) server_negotiation.cc:407] Connection header received
0924 10:00:59.518698 (+ 1533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:00:59.518718 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:00:59.519106 (+ 388us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:00:59.519459 (+ 353us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:00:59.521016 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:00:59.522275 (+ 1259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:00:59.524046 (+ 1771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:00:59.524692 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:00:59.524812 (+ 120us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:00:59.529532 (+ 4720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:00:59.529541 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:00:59.529554 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:00:59.529590 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:00:59.532532 (+ 2942us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:00:59.533447 (+ 915us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:00:59.533453 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:00:59.533459 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:00:59.533635 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:00:59.534083 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:00:59.534087 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:00:59.534089 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:00:59.534634 (+ 545us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:00:59.534889 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0924 10:00:59.535427 (+ 538us) server_negotiation.cc:299] Negotiation successful
0924 10:00:59.535756 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":513,"thread_start_us":314,"threads_started":1}
I20260924 10:00:59.540298 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 10:00:59.541076 21831 raft_consensus.cc:348] T 00000000000000000000000000000000 P 40023ae58d5e494381581fc98d1b056b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40023ae58d5e494381581fc98d1b056b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41091 } }
I20260924 10:00:59.541467 21831 raft_consensus.cc:374] T 00000000000000000000000000000000 P 40023ae58d5e494381581fc98d1b056b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:00:59.541565 21831 raft_consensus.cc:729] T 00000000000000000000000000000000 P 40023ae58d5e494381581fc98d1b056b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40023ae58d5e494381581fc98d1b056b, State: Initialized, Role: FOLLOWER
I20260924 10:00:59.542122 21831 consensus_queue.cc:260] T 00000000000000000000000000000000 P 40023ae58d5e494381581fc98d1b056b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40023ae58d5e494381581fc98d1b056b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41091 } }
I20260924 10:00:59.542366 21831 raft_consensus.cc:388] T 00000000000000000000000000000000 P 40023ae58d5e494381581fc98d1b056b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:00:59.542493 21831 raft_consensus.cc:482] T 00000000000000000000000000000000 P 40023ae58d5e494381581fc98d1b056b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:00:59.542657 21831 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 40023ae58d5e494381581fc98d1b056b [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:00:59.545392 21831 raft_consensus.cc:504] T 00000000000000000000000000000000 P 40023ae58d5e494381581fc98d1b056b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40023ae58d5e494381581fc98d1b056b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41091 } }
I20260924 10:00:59.545888 21831 leader_election.cc:302] T 00000000000000000000000000000000 P 40023ae58d5e494381581fc98d1b056b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 40023ae58d5e494381581fc98d1b056b; no voters:
I20260924 10:00:59.546465 21831 leader_election.cc:288] T 00000000000000000000000000000000 P 40023ae58d5e494381581fc98d1b056b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:00:59.546607 21841 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 40023ae58d5e494381581fc98d1b056b [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:00:59.547868 21841 raft_consensus.cc:686] T 00000000000000000000000000000000 P 40023ae58d5e494381581fc98d1b056b [term 1 LEADER]: Becoming Leader. State: Replica: 40023ae58d5e494381581fc98d1b056b, State: Running, Role: LEADER
I20260924 10:00:59.548403 21841 consensus_queue.cc:237] T 00000000000000000000000000000000 P 40023ae58d5e494381581fc98d1b056b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40023ae58d5e494381581fc98d1b056b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41091 } }
I20260924 10:00:59.548583 21831 sys_catalog.cc:564] T 00000000000000000000000000000000 P 40023ae58d5e494381581fc98d1b056b [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:00:59.557528 21843 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40023ae58d5e494381581fc98d1b056b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 40023ae58d5e494381581fc98d1b056b. Latest consensus state: current_term: 1 leader_uuid: "40023ae58d5e494381581fc98d1b056b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40023ae58d5e494381581fc98d1b056b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41091 } } }
I20260924 10:00:59.558022 21843 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40023ae58d5e494381581fc98d1b056b [sys.catalog]: This master's current role is: LEADER
I20260924 10:00:59.559335 21842 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40023ae58d5e494381581fc98d1b056b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "40023ae58d5e494381581fc98d1b056b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40023ae58d5e494381581fc98d1b056b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41091 } } }
I20260924 10:00:59.559615 21842 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40023ae58d5e494381581fc98d1b056b [sys.catalog]: This master's current role is: LEADER
I20260924 10:00:59.561862 21834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:00:59.545063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.162:49067 (local address 127.0.0.1:41091)
0924 10:00:59.545224 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0924 10:00:59.545230 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:00:59.545626 (+ 396us) server_negotiation.cc:407] Connection header received
0924 10:00:59.545753 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:00:59.545757 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:00:59.545827 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:00:59.545937 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:00:59.547630 (+ 1693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:00:59.548805 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:00:59.550557 (+ 1752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:00:59.551223 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:00:59.551376 (+ 153us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:00:59.554416 (+ 3040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:00:59.554423 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:00:59.554429 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:00:59.554487 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:00:59.558083 (+ 3596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:00:59.559176 (+ 1093us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:00:59.559182 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:00:59.559187 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:00:59.559350 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:00:59.559901 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:00:59.559907 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:00:59.559911 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:00:59.560336 (+ 425us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:00:59.560588 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0924 10:00:59.561287 (+ 699us) server_negotiation.cc:299] Negotiation successful
0924 10:00:59.561474 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":49}
I20260924 10:00:59.565948 21850 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:00:59.570892 21850 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:00:59.579445 21850 catalog_manager.cc:1347] Generated new cluster ID: ac53acb594874da48384226e23c37a66
I20260924 10:00:59.579538 21850 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:00:59.591256 21850 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:00:59.592952 21850 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:00:59.604775 21850 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 40023ae58d5e494381581fc98d1b056b: Generated new TSK 0
I20260924 10:00:59.605718 21850 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 10:00:59.611300 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu pid 21764
I20260924 10:00:59.612249 21826 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:00:59.698827 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu with pid 21764
W20260924 10:00:59.712244 21749 connection.cc:535] client connection to 127.0.0.1:41091 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41091): Connection reset by peer (error 104)
2026-09-24T10:00:59Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1812 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790243735634835-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:00:59 dist-test-slave-w8d2 krb5kdc[21863](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:00:59 dist-test-slave-w8d2 krb5kdc[21863](info): set up 2 sockets
Sep 24 10:00:59 dist-test-slave-w8d2 krb5kdc[21863](info): commencing operation
krb5kdc: starting...
W20260924 10:01:00.703480 388 mini_kdc.cc:121] Time spent starting KDC: real 0.973s user 0.000s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790243735634835-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790243735634835-388-0/krb5kdc/test-user.keytab.
Sep 24 10:01:00 dist-test-slave-w8d2 krb5kdc[21863](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244060, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:01:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:01:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790243735634835-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790243735634835-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 10:01:00.929440 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
/tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790243735634835-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790243735634835-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790243735634835-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790243735634835-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41527
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790243735634835-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41527
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790243735634835-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790243735634835-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790243735634835-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:01:01.239323 21876 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:01:01.239629 21876 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:01:01.239760 21876 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:01:01.246872 21876 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:01:01.246997 21876 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:01:01.247076 21876 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:01:01.247157 21876 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:01:01.247215 21876 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:01:01.257036 21876 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790243735634835-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790243735634835-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41527
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790243735634835-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41527
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790243735634835-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790243735634835-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:01:01.258301 21876 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:01:01.259565 21876 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:01:01.270519 21881 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:01:01.270495 21884 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:01:01.270483 21882 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:01:01.271782 21876 server_base.cc:1034] running on GCE node
I20260924 10:01:01.272478 21876 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:01:01.273797 21876 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:01:01.274988 21876 hybrid_clock.cc:648] HybridClock initialized: now 1790244061274951 us; error 26 us; skew 500 ppm
I20260924 10:01:01.275350 21876 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:01:01 dist-test-slave-w8d2 krb5kdc[21863](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244061, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:01:01.278134 21876 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 10:01:01.279531 21876 webserver.cc:460] Webserver started at http://127.0.0.1:42625/ using document root <none> and password file <none>
I20260924 10:01:01.280155 21876 fs_manager.cc:359] Metadata directory not provided
I20260924 10:01:01.280263 21876 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:01:01.280522 21876 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:01:01.283290 21876 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790243735634835-388-0/minicluster-data/master-0/data/instance:
uuid: "0b5b151ac9fc4db2b0363cee2398305d"
format_stamp: "Formatted at 2026-09-24 10:01:01 on dist-test-slave-w8d2"
server_key: "e4d01aaa2c60b12925a4f3f8d4754f1b"
server_key_iv: "cf3e2909609b351b8c92e220db380131"
server_key_version: "encryptionkey@0"
I20260924 10:01:01.283979 21876 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790243735634835-388-0/minicluster-data/master-0/wal/instance:
uuid: "0b5b151ac9fc4db2b0363cee2398305d"
format_stamp: "Formatted at 2026-09-24 10:01:01 on dist-test-slave-w8d2"
server_key: "e4d01aaa2c60b12925a4f3f8d4754f1b"
server_key_iv: "cf3e2909609b351b8c92e220db380131"
server_key_version: "encryptionkey@0"
I20260924 10:01:01.288614 21876 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260924 10:01:01.291747 21891 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:01:01.292910 21876 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 10:01:01.293048 21876 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790243735634835-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790243735634835-388-0/minicluster-data/master-0/wal
uuid: "0b5b151ac9fc4db2b0363cee2398305d"
format_stamp: "Formatted at 2026-09-24 10:01:01 on dist-test-slave-w8d2"
server_key: "e4d01aaa2c60b12925a4f3f8d4754f1b"
server_key_iv: "cf3e2909609b351b8c92e220db380131"
server_key_version: "encryptionkey@0"
I20260924 10:01:01.293215 21876 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790243735634835-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790243735634835-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790243735634835-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:01:01.310460 21876 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:01:01.314116 21876 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:01:01.314360 21876 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:01:01.330567 21876 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41527
I20260924 10:01:01.330607 21942 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41527 every 8 connection(s)
I20260924 10:01:01.331912 21876 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790243735634835-388-0/minicluster-data/master-0/data/info.pb
I20260924 10:01:01.335495 21943 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:01:01.342176 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu as pid 21876
I20260924 10:01:01.342344 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790243735634835-388-0/minicluster-data/master-0/wal/instance
I20260924 10:01:01.342830 388 external_mini_cluster.cc:1428] Setting key cefa3080064a9b030f8ed9d2fe5f6531
I20260924 10:01:01.345964 21943 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b5b151ac9fc4db2b0363cee2398305d: Bootstrap starting.
I20260924 10:01:01.349109 21943 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0b5b151ac9fc4db2b0363cee2398305d: Neither blocks nor log segments found. Creating new log.
I20260924 10:01:01.350235 21943 log.cc:824] T 00000000000000000000000000000000 P 0b5b151ac9fc4db2b0363cee2398305d: Log is configured to *not* fsync() on all Append() calls
I20260924 10:01:01.353214 21943 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b5b151ac9fc4db2b0363cee2398305d: No bootstrap required, opened a new log
I20260924 10:01:01.355383 21946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:01.344560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.162:46537 (local address 127.0.0.1:41527)
0924 10:01:01.345224 (+ 664us) server_negotiation.cc:206] Beginning negotiation
0924 10:01:01.345236 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 10:01:01.345275 (+ 39us) server_negotiation.cc:407] Connection header received
0924 10:01:01.346944 (+ 1669us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:01:01.346965 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:01:01.347392 (+ 427us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:01:01.347711 (+ 319us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:01:01.348236 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:01.349495 (+ 1259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:01.351207 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:01.351936 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:01.352054 (+ 118us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:01:01.353246 (+ 1192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:01:01.353253 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:01:01.353265 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:01:01.353788 (+ 523us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:01:01.353866 (+ 78us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:01:01.353889 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:01:01.353997 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0924 10:01:01.354241 (+ 244us) server_negotiation.cc:299] Negotiation successful
0924 10:01:01.354395 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":476,"thread_start_us":287,"threads_started":1}
I20260924 10:01:01.358781 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 10:01:01.361649 21943 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0b5b151ac9fc4db2b0363cee2398305d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b5b151ac9fc4db2b0363cee2398305d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41527 } }
I20260924 10:01:01.362084 21943 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0b5b151ac9fc4db2b0363cee2398305d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:01:01.362196 21943 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0b5b151ac9fc4db2b0363cee2398305d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b5b151ac9fc4db2b0363cee2398305d, State: Initialized, Role: FOLLOWER
I20260924 10:01:01.362835 21943 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0b5b151ac9fc4db2b0363cee2398305d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b5b151ac9fc4db2b0363cee2398305d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41527 } }
I20260924 10:01:01.363118 21943 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0b5b151ac9fc4db2b0363cee2398305d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:01:01.363268 21943 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0b5b151ac9fc4db2b0363cee2398305d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:01:01.363435 21943 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0b5b151ac9fc4db2b0363cee2398305d [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:01:01.366228 21943 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0b5b151ac9fc4db2b0363cee2398305d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b5b151ac9fc4db2b0363cee2398305d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41527 } }
I20260924 10:01:01.366695 21943 leader_election.cc:302] T 00000000000000000000000000000000 P 0b5b151ac9fc4db2b0363cee2398305d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0b5b151ac9fc4db2b0363cee2398305d; no voters:
I20260924 10:01:01.367210 21943 leader_election.cc:288] T 00000000000000000000000000000000 P 0b5b151ac9fc4db2b0363cee2398305d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:01:01.367352 21953 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0b5b151ac9fc4db2b0363cee2398305d [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:01:01.367903 21953 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0b5b151ac9fc4db2b0363cee2398305d [term 1 LEADER]: Becoming Leader. State: Replica: 0b5b151ac9fc4db2b0363cee2398305d, State: Running, Role: LEADER
I20260924 10:01:01.368521 21953 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0b5b151ac9fc4db2b0363cee2398305d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b5b151ac9fc4db2b0363cee2398305d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41527 } }
I20260924 10:01:01.370288 21953 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b5b151ac9fc4db2b0363cee2398305d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0b5b151ac9fc4db2b0363cee2398305d. Latest consensus state: current_term: 1 leader_uuid: "0b5b151ac9fc4db2b0363cee2398305d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b5b151ac9fc4db2b0363cee2398305d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41527 } } }
I20260924 10:01:01.370944 21946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:01.363707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.162:57169 (local address 127.0.0.1:41527)
0924 10:01:01.363865 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0924 10:01:01.363871 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:01:01.364361 (+ 490us) server_negotiation.cc:407] Connection header received
0924 10:01:01.364510 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:01:01.364515 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:01:01.364583 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:01:01.364696 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:01:01.365081 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:01.366189 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:01.367827 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:01.368473 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:01.368579 (+ 106us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:01:01.370042 (+ 1463us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:01:01.370048 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:01:01.370052 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:01:01.370076 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:01:01.370144 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:01:01.370168 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:01:01.370250 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0924 10:01:01.370580 (+ 330us) server_negotiation.cc:299] Negotiation successful
0924 10:01:01.370632 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 10:01:01.371773 21953 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b5b151ac9fc4db2b0363cee2398305d [sys.catalog]: This master's current role is: LEADER
I20260924 10:01:01.372781 21955 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b5b151ac9fc4db2b0363cee2398305d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0b5b151ac9fc4db2b0363cee2398305d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b5b151ac9fc4db2b0363cee2398305d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41527 } } }
I20260924 10:01:01.373162 21955 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b5b151ac9fc4db2b0363cee2398305d [sys.catalog]: This master's current role is: LEADER
I20260924 10:01:01.374672 21943 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0b5b151ac9fc4db2b0363cee2398305d [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:01:01.374858 21957 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:01:01.379673 21957 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:01:01.388000 21957 catalog_manager.cc:1347] Generated new cluster ID: 427a165ec7d84e9ea742904a8f1e5427
I20260924 10:01:01.388108 21957 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:01:01.427986 21957 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:01:01.429766 21957 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:01:01.436717 21957 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0b5b151ac9fc4db2b0363cee2398305d: Generated new TSK 0
I20260924 10:01:01.437678 21957 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 10:01:01.449891 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu pid 21876
W20260924 10:01:01.450975 21938 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.162:46537
W20260924 10:01:01.451658 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu pid 21876: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 10:01:01.451800 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu with pid 21876
W20260924 10:01:01.465314 21861 connection.cc:535] client connection to 127.0.0.1:41527 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41527): Connection reset by peer (error 104)
2026-09-24T10:01:01Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1752 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790243735634835-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:01:01 dist-test-slave-w8d2 krb5kdc[21974](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:01:01 dist-test-slave-w8d2 krb5kdc[21974](info): set up 2 sockets
Sep 24 10:01:01 dist-test-slave-w8d2 krb5kdc[21974](info): commencing operation
krb5kdc: starting...
W20260924 10:01:02.451715 388 mini_kdc.cc:121] Time spent starting KDC: real 0.968s user 0.008s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790243735634835-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790243735634835-388-0/krb5kdc/test-user.keytab.
Sep 24 10:01:02 dist-test-slave-w8d2 krb5kdc[21974](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244062, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:01:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:01:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790243735634835-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790243735634835-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 10:01:02.679064 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
/tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790243735634835-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790243735634835-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790243735634835-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790243735634835-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36275
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790243735634835-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36275
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790243735634835-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790243735634835-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790243735634835-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:01:02.997692 21987 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:01:02.997983 21987 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:01:02.998072 21987 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:01:03.005179 21987 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:01:03.005281 21987 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:01:03.005333 21987 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:01:03.005390 21987 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:01:03.005441 21987 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:01:03.014991 21987 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790243735634835-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790243735634835-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36275
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790243735634835-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36275
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790243735634835-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790243735634835-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:01:03.016252 21987 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:01:03.017453 21987 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:01:03.028698 21995 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:01:03.028723 21992 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:01:03.028687 21993 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:01:03.029289 21987 server_base.cc:1034] running on GCE node
I20260924 10:01:03.030125 21987 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:01:03.031514 21987 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:01:03.032716 21987 hybrid_clock.cc:648] HybridClock initialized: now 1790244063032668 us; error 36 us; skew 500 ppm
I20260924 10:01:03.033043 21987 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:01:03 dist-test-slave-w8d2 krb5kdc[21974](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244063, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:01:03.040004 21987 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 10:01:03.041456 21987 webserver.cc:460] Webserver started at http://127.0.0.1:46121/ using document root <none> and password file <none>
I20260924 10:01:03.042127 21987 fs_manager.cc:359] Metadata directory not provided
I20260924 10:01:03.042224 21987 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:01:03.042484 21987 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:01:03.045351 21987 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790243735634835-388-0/minicluster-data/master-0/data/instance:
uuid: "c7924a0fd43c4845b490305a02c8daff"
format_stamp: "Formatted at 2026-09-24 10:01:03 on dist-test-slave-w8d2"
server_key: "3566db542fe083664147adc6abc0c978"
server_key_iv: "498311f5bfe17e3a1e68aa93cd376a93"
server_key_version: "encryptionkey@0"
I20260924 10:01:03.046030 21987 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790243735634835-388-0/minicluster-data/master-0/wal/instance:
uuid: "c7924a0fd43c4845b490305a02c8daff"
format_stamp: "Formatted at 2026-09-24 10:01:03 on dist-test-slave-w8d2"
server_key: "3566db542fe083664147adc6abc0c978"
server_key_iv: "498311f5bfe17e3a1e68aa93cd376a93"
server_key_version: "encryptionkey@0"
I20260924 10:01:03.050923 21987 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260924 10:01:03.054291 22002 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:01:03.055723 21987 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260924 10:01:03.055883 21987 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790243735634835-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790243735634835-388-0/minicluster-data/master-0/wal
uuid: "c7924a0fd43c4845b490305a02c8daff"
format_stamp: "Formatted at 2026-09-24 10:01:03 on dist-test-slave-w8d2"
server_key: "3566db542fe083664147adc6abc0c978"
server_key_iv: "498311f5bfe17e3a1e68aa93cd376a93"
server_key_version: "encryptionkey@0"
I20260924 10:01:03.056075 21987 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790243735634835-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790243735634835-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790243735634835-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:01:03.061455 21987 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:01:03.065145 21987 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:01:03.065392 21987 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:01:03.081372 21987 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36275
I20260924 10:01:03.081413 22053 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36275 every 8 connection(s)
I20260924 10:01:03.082672 21987 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790243735634835-388-0/minicluster-data/master-0/data/info.pb
I20260924 10:01:03.086237 22054 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:01:03.092177 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu as pid 21987
I20260924 10:01:03.092355 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790243735634835-388-0/minicluster-data/master-0/wal/instance
I20260924 10:01:03.092845 388 external_mini_cluster.cc:1428] Setting key 1f4cf17e05caa94c6b6d87ec81eae352
I20260924 10:01:03.097005 22054 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c7924a0fd43c4845b490305a02c8daff: Bootstrap starting.
I20260924 10:01:03.100165 22054 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c7924a0fd43c4845b490305a02c8daff: Neither blocks nor log segments found. Creating new log.
I20260924 10:01:03.100199 22057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:03.094587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.162:53895 (local address 127.0.0.1:36275)
0924 10:01:03.095418 (+ 831us) server_negotiation.cc:206] Beginning negotiation
0924 10:01:03.095429 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 10:01:03.095466 (+ 37us) server_negotiation.cc:407] Connection header received
0924 10:01:03.097054 (+ 1588us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:01:03.097080 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:01:03.097419 (+ 339us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:01:03.097747 (+ 328us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:01:03.098696 (+ 949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:01:03.098702 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:01:03.098713 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:01:03.098728 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:01:03.098791 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:01:03.098811 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:01:03.099002 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0924 10:01:03.099129 (+ 127us) server_negotiation.cc:299] Negotiation successful
0924 10:01:03.099339 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":599,"thread_start_us":346,"threads_started":1}
I20260924 10:01:03.101231 22054 log.cc:824] T 00000000000000000000000000000000 P c7924a0fd43c4845b490305a02c8daff: Log is configured to *not* fsync() on all Append() calls
I20260924 10:01:03.103721 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 10:01:03.104120 22054 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c7924a0fd43c4845b490305a02c8daff: No bootstrap required, opened a new log
I20260924 10:01:03.109987 22057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:03.107859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.162:59507 (local address 127.0.0.1:36275)
0924 10:01:03.108038 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0924 10:01:03.108044 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:01:03.108269 (+ 225us) server_negotiation.cc:407] Connection header received
0924 10:01:03.108395 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:01:03.108401 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:01:03.108459 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:01:03.108571 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:01:03.109377 (+ 806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:01:03.109383 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:01:03.109388 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:01:03.109407 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:01:03.109451 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:01:03.109471 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:01:03.109592 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0924 10:01:03.109637 (+ 45us) server_negotiation.cc:299] Negotiation successful
0924 10:01:03.109692 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260924 10:01:03.112178 22054 raft_consensus.cc:348] T 00000000000000000000000000000000 P c7924a0fd43c4845b490305a02c8daff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7924a0fd43c4845b490305a02c8daff" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36275 } }
I20260924 10:01:03.112520 22054 raft_consensus.cc:374] T 00000000000000000000000000000000 P c7924a0fd43c4845b490305a02c8daff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:01:03.112599 22054 raft_consensus.cc:729] T 00000000000000000000000000000000 P c7924a0fd43c4845b490305a02c8daff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7924a0fd43c4845b490305a02c8daff, State: Initialized, Role: FOLLOWER
I20260924 10:01:03.113090 22054 consensus_queue.cc:260] T 00000000000000000000000000000000 P c7924a0fd43c4845b490305a02c8daff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7924a0fd43c4845b490305a02c8daff" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36275 } }
I20260924 10:01:03.113289 22054 raft_consensus.cc:388] T 00000000000000000000000000000000 P c7924a0fd43c4845b490305a02c8daff [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:01:03.113384 22054 raft_consensus.cc:482] T 00000000000000000000000000000000 P c7924a0fd43c4845b490305a02c8daff [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:01:03.113514 22054 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c7924a0fd43c4845b490305a02c8daff [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:01:03.115881 22054 raft_consensus.cc:504] T 00000000000000000000000000000000 P c7924a0fd43c4845b490305a02c8daff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7924a0fd43c4845b490305a02c8daff" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36275 } }
I20260924 10:01:03.116320 22054 leader_election.cc:302] T 00000000000000000000000000000000 P c7924a0fd43c4845b490305a02c8daff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c7924a0fd43c4845b490305a02c8daff; no voters:
I20260924 10:01:03.116844 22054 leader_election.cc:288] T 00000000000000000000000000000000 P c7924a0fd43c4845b490305a02c8daff [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:01:03.116952 22064 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c7924a0fd43c4845b490305a02c8daff [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:01:03.117626 22064 raft_consensus.cc:686] T 00000000000000000000000000000000 P c7924a0fd43c4845b490305a02c8daff [term 1 LEADER]: Becoming Leader. State: Replica: c7924a0fd43c4845b490305a02c8daff, State: Running, Role: LEADER
I20260924 10:01:03.118161 22064 consensus_queue.cc:237] T 00000000000000000000000000000000 P c7924a0fd43c4845b490305a02c8daff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7924a0fd43c4845b490305a02c8daff" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36275 } }
I20260924 10:01:03.118741 22054 sys_catalog.cc:564] T 00000000000000000000000000000000 P c7924a0fd43c4845b490305a02c8daff [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:01:03.124330 22066 sys_catalog.cc:455] T 00000000000000000000000000000000 P c7924a0fd43c4845b490305a02c8daff [sys.catalog]: SysCatalogTable state changed. Reason: New leader c7924a0fd43c4845b490305a02c8daff. Latest consensus state: current_term: 1 leader_uuid: "c7924a0fd43c4845b490305a02c8daff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7924a0fd43c4845b490305a02c8daff" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36275 } } }
I20260924 10:01:03.124728 22066 sys_catalog.cc:458] T 00000000000000000000000000000000 P c7924a0fd43c4845b490305a02c8daff [sys.catalog]: This master's current role is: LEADER
I20260924 10:01:03.124795 22065 sys_catalog.cc:455] T 00000000000000000000000000000000 P c7924a0fd43c4845b490305a02c8daff [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c7924a0fd43c4845b490305a02c8daff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7924a0fd43c4845b490305a02c8daff" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36275 } } }
I20260924 10:01:03.125108 22065 sys_catalog.cc:458] T 00000000000000000000000000000000 P c7924a0fd43c4845b490305a02c8daff [sys.catalog]: This master's current role is: LEADER
W20260924 10:01:03.130177 22078 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c7924a0fd43c4845b490305a02c8daff: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 10:01:03.130347 22078 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 10:01:03.130584 22079 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:01:03.132759 22079 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:01:03.140658 22079 catalog_manager.cc:1347] Generated new cluster ID: f81d2ede7bda4ca2bacdd90857c11bf3
I20260924 10:01:03.140763 22079 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:01:03.167932 22079 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:01:03.168452 22079 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:01:03.185267 22079 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c7924a0fd43c4845b490305a02c8daff: Generated new TSK 0
I20260924 10:01:03.186136 22079 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 10:01:03.200445 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu pid 21987
W20260924 10:01:03.201489 22049 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.162:53895
W20260924 10:01:03.202116 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu pid 21987: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 10:01:03.202235 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu with pid 21987
2026-09-24T10:01:03Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1754 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790243735634835-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:01:03 dist-test-slave-w8d2 krb5kdc[22085](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:01:03 dist-test-slave-w8d2 krb5kdc[22085](info): set up 2 sockets
Sep 24 10:01:03 dist-test-slave-w8d2 krb5kdc[22085](info): commencing operation
krb5kdc: starting...
W20260924 10:01:04.207372 388 mini_kdc.cc:121] Time spent starting KDC: real 0.969s user 0.000s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790243735634835-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790243735634835-388-0/krb5kdc/test-user.keytab.
Sep 24 10:01:04 dist-test-slave-w8d2 krb5kdc[22085](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244064, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:01:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:01:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790243735634835-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790243735634835-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260924 10:01:04.437865 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
/tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790243735634835-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790243735634835-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790243735634835-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790243735634835-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36101
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790243735634835-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:36101
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790243735634835-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790243735634835-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790243735634835-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:01:04.759807 22098 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:01:04.760066 22098 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:01:04.760164 22098 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:01:04.767153 22098 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:01:04.767280 22098 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:01:04.767344 22098 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:01:04.767426 22098 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:01:04.767480 22098 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:01:04.777094 22098 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790243735634835-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790243735634835-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:36101
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790243735634835-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36101
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790243735634835-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790243735634835-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:01:04.778306 22098 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:01:04.779590 22098 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:01:04.790305 22103 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:01:04.790324 22106 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:01:04.790373 22104 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:01:04.791008 22098 server_base.cc:1034] running on GCE node
I20260924 10:01:04.791733 22098 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:01:04.792990 22098 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:01:04.794193 22098 hybrid_clock.cc:648] HybridClock initialized: now 1790244064794132 us; error 38 us; skew 500 ppm
I20260924 10:01:04.794517 22098 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:01:04 dist-test-slave-w8d2 krb5kdc[22085](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244064, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:01:04.797366 22098 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260924 10:01:04.798763 22098 webserver.cc:460] Webserver started at http://127.0.97.62:46437/ using document root <none> and password file <none>
I20260924 10:01:04.799417 22098 fs_manager.cc:359] Metadata directory not provided
I20260924 10:01:04.799525 22098 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:01:04.799804 22098 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:01:04.802593 22098 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790243735634835-388-0/minicluster-data/master-0/data/instance:
uuid: "88cf0ad05ae94c5a9742a179b40b9b7a"
format_stamp: "Formatted at 2026-09-24 10:01:04 on dist-test-slave-w8d2"
server_key: "5467eabd3d93f840d197e9d8897346fe"
server_key_iv: "30d05b94e103800dfb43dd63ec340d45"
server_key_version: "encryptionkey@0"
I20260924 10:01:04.803328 22098 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790243735634835-388-0/minicluster-data/master-0/wal/instance:
uuid: "88cf0ad05ae94c5a9742a179b40b9b7a"
format_stamp: "Formatted at 2026-09-24 10:01:04 on dist-test-slave-w8d2"
server_key: "5467eabd3d93f840d197e9d8897346fe"
server_key_iv: "30d05b94e103800dfb43dd63ec340d45"
server_key_version: "encryptionkey@0"
I20260924 10:01:04.808360 22098 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260924 10:01:04.811504 22113 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:01:04.812741 22098 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260924 10:01:04.812887 22098 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790243735634835-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790243735634835-388-0/minicluster-data/master-0/wal
uuid: "88cf0ad05ae94c5a9742a179b40b9b7a"
format_stamp: "Formatted at 2026-09-24 10:01:04 on dist-test-slave-w8d2"
server_key: "5467eabd3d93f840d197e9d8897346fe"
server_key_iv: "30d05b94e103800dfb43dd63ec340d45"
server_key_version: "encryptionkey@0"
I20260924 10:01:04.813050 22098 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790243735634835-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790243735634835-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790243735634835-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:01:04.830935 22098 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:01:04.834537 22098 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:01:04.834784 22098 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:01:04.850520 22098 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:36101
I20260924 10:01:04.850540 22164 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:36101 every 8 connection(s)
I20260924 10:01:04.851794 22098 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790243735634835-388-0/minicluster-data/master-0/data/info.pb
I20260924 10:01:04.855144 22165 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:01:04.859107 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu as pid 22098
I20260924 10:01:04.859328 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790243735634835-388-0/minicluster-data/master-0/wal/instance
I20260924 10:01:04.859861 388 external_mini_cluster.cc:1428] Setting key 7e4dc09717b9d26afbbdc3f2a3596cd4
I20260924 10:01:04.865721 22165 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88cf0ad05ae94c5a9742a179b40b9b7a: Bootstrap starting.
I20260924 10:01:04.869840 22165 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 88cf0ad05ae94c5a9742a179b40b9b7a: Neither blocks nor log segments found. Creating new log.
I20260924 10:01:04.871073 22165 log.cc:824] T 00000000000000000000000000000000 P 88cf0ad05ae94c5a9742a179b40b9b7a: Log is configured to *not* fsync() on all Append() calls
Sep 24 10:01:04 dist-test-slave-w8d2 krb5kdc[22085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244064, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 10:01:04.875650 22165 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88cf0ad05ae94c5a9742a179b40b9b7a: No bootstrap required, opened a new log
I20260924 10:01:04.883023 22168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:04.861434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37035 (local address 127.0.97.62:36101)
0924 10:01:04.862110 (+ 676us) server_negotiation.cc:206] Beginning negotiation
0924 10:01:04.862145 (+ 35us) server_negotiation.cc:399] Waiting for connection header
0924 10:01:04.862183 (+ 38us) server_negotiation.cc:407] Connection header received
0924 10:01:04.864355 (+ 2172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:01:04.864375 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:01:04.864728 (+ 353us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:01:04.865024 (+ 296us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:01:04.866694 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:04.868050 (+ 1356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:04.869809 (+ 1759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:04.870791 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:04.870968 (+ 177us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:01:04.874866 (+ 3898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:01:04.874877 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:01:04.874894 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:01:04.874936 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:01:04.878621 (+ 3685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:01:04.879389 (+ 768us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:01:04.879401 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:01:04.879413 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:01:04.879625 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:01:04.879991 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:01:04.879996 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:01:04.879999 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:01:04.880666 (+ 667us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:01:04.880984 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0924 10:01:04.881320 (+ 336us) server_negotiation.cc:299] Negotiation successful
0924 10:01:04.881671 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":478,"thread_start_us":306,"threads_started":1}
I20260924 10:01:04.885712 22165 raft_consensus.cc:348] T 00000000000000000000000000000000 P 88cf0ad05ae94c5a9742a179b40b9b7a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88cf0ad05ae94c5a9742a179b40b9b7a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36101 } }
I20260924 10:01:04.886089 22165 raft_consensus.cc:374] T 00000000000000000000000000000000 P 88cf0ad05ae94c5a9742a179b40b9b7a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:01:04.886196 22165 raft_consensus.cc:729] T 00000000000000000000000000000000 P 88cf0ad05ae94c5a9742a179b40b9b7a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88cf0ad05ae94c5a9742a179b40b9b7a, State: Initialized, Role: FOLLOWER
I20260924 10:01:04.886785 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 10:01:04.886766 22165 consensus_queue.cc:260] T 00000000000000000000000000000000 P 88cf0ad05ae94c5a9742a179b40b9b7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88cf0ad05ae94c5a9742a179b40b9b7a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36101 } }
I20260924 10:01:04.887015 22165 raft_consensus.cc:388] T 00000000000000000000000000000000 P 88cf0ad05ae94c5a9742a179b40b9b7a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:01:04.887140 22165 raft_consensus.cc:482] T 00000000000000000000000000000000 P 88cf0ad05ae94c5a9742a179b40b9b7a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:01:04.887307 22165 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 88cf0ad05ae94c5a9742a179b40b9b7a [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:01:04.889423 22165 raft_consensus.cc:504] T 00000000000000000000000000000000 P 88cf0ad05ae94c5a9742a179b40b9b7a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88cf0ad05ae94c5a9742a179b40b9b7a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36101 } }
I20260924 10:01:04.889899 22165 leader_election.cc:302] T 00000000000000000000000000000000 P 88cf0ad05ae94c5a9742a179b40b9b7a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 88cf0ad05ae94c5a9742a179b40b9b7a; no voters:
I20260924 10:01:04.890483 22165 leader_election.cc:288] T 00000000000000000000000000000000 P 88cf0ad05ae94c5a9742a179b40b9b7a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:01:04.890586 22174 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 88cf0ad05ae94c5a9742a179b40b9b7a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:01:04.892835 22165 sys_catalog.cc:564] T 00000000000000000000000000000000 P 88cf0ad05ae94c5a9742a179b40b9b7a [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:01:04.895004 22174 raft_consensus.cc:686] T 00000000000000000000000000000000 P 88cf0ad05ae94c5a9742a179b40b9b7a [term 1 LEADER]: Becoming Leader. State: Replica: 88cf0ad05ae94c5a9742a179b40b9b7a, State: Running, Role: LEADER
I20260924 10:01:04.895678 22174 consensus_queue.cc:237] T 00000000000000000000000000000000 P 88cf0ad05ae94c5a9742a179b40b9b7a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88cf0ad05ae94c5a9742a179b40b9b7a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36101 } }
I20260924 10:01:04.898453 22175 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88cf0ad05ae94c5a9742a179b40b9b7a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "88cf0ad05ae94c5a9742a179b40b9b7a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88cf0ad05ae94c5a9742a179b40b9b7a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36101 } } }
I20260924 10:01:04.898818 22175 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88cf0ad05ae94c5a9742a179b40b9b7a [sys.catalog]: This master's current role is: LEADER
I20260924 10:01:04.903493 22176 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88cf0ad05ae94c5a9742a179b40b9b7a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88cf0ad05ae94c5a9742a179b40b9b7a. Latest consensus state: current_term: 1 leader_uuid: "88cf0ad05ae94c5a9742a179b40b9b7a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88cf0ad05ae94c5a9742a179b40b9b7a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36101 } } }
I20260924 10:01:04.904280 22176 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88cf0ad05ae94c5a9742a179b40b9b7a [sys.catalog]: This master's current role is: LEADER
I20260924 10:01:04.907373 22181 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:01:04.911820 22168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:04.890771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47129 (local address 127.0.97.62:36101)
0924 10:01:04.890942 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0924 10:01:04.890948 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:01:04.894479 (+ 3531us) server_negotiation.cc:407] Connection header received
0924 10:01:04.894639 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:01:04.894645 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:01:04.894719 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:01:04.894811 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:01:04.896449 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:04.897243 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:04.899672 (+ 2429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:04.900406 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:04.900510 (+ 104us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:01:04.903942 (+ 3432us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:01:04.903950 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:01:04.903955 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:01:04.903996 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:01:04.907770 (+ 3774us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:01:04.908669 (+ 899us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:01:04.908678 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:01:04.908682 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:01:04.908815 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:01:04.909314 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:01:04.909324 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:01:04.909327 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:01:04.909803 (+ 476us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:01:04.910081 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0924 10:01:04.911221 (+ 1140us) server_negotiation.cc:299] Negotiation successful
0924 10:01:04.911430 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 10:01:04.912366 22181 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:01:04.922269 22181 catalog_manager.cc:1347] Generated new cluster ID: 6fb8e8b2044d4020a8503ed57ee2c257
I20260924 10:01:04.922395 22181 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:01:04.938891 22181 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:01:04.940429 22181 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:01:04.953542 22181 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 88cf0ad05ae94c5a9742a179b40b9b7a: Generated new TSK 0
I20260924 10:01:04.954334 22181 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 10:01:04.962813 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu pid 22098
I20260924 10:01:04.964232 22160 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:01:05.054451 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu with pid 22098
W20260924 10:01:05.068005 22083 connection.cc:535] client connection to 127.0.97.62:36101 recv error: Network error: failed to read from TLS socket (remote: 127.0.97.62:36101): Connection reset by peer (error 104)
2026-09-24T10:01:05Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1848 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790243735634835-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:01:05 dist-test-slave-w8d2 krb5kdc[22197](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:01:05 dist-test-slave-w8d2 krb5kdc[22197](info): set up 2 sockets
Sep 24 10:01:05 dist-test-slave-w8d2 krb5kdc[22197](info): commencing operation
krb5kdc: starting...
W20260924 10:01:06.067440 388 mini_kdc.cc:121] Time spent starting KDC: real 0.981s user 0.005s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790243735634835-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790243735634835-388-0/krb5kdc/test-user.keytab.
Sep 24 10:01:06 dist-test-slave-w8d2 krb5kdc[22197](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244066, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:01:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:01:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790243735634835-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790243735634835-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260924 10:01:06.292145 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
/tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790243735634835-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790243735634835-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790243735634835-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790243735634835-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35835
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790243735634835-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:35835
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790243735634835-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790243735634835-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790243735634835-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:01:06.603751 22210 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:01:06.604023 22210 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:01:06.604113 22210 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:01:06.611168 22210 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:01:06.611289 22210 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:01:06.611341 22210 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:01:06.611402 22210 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:01:06.611444 22210 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:01:06.621059 22210 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790243735634835-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790243735634835-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:35835
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790243735634835-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35835
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790243735634835-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790243735634835-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:01:06.622277 22210 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:01:06.623575 22210 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:01:06.634584 22216 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:01:06.634649 22215 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:01:06.634629 22218 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:01:06.635380 22210 server_base.cc:1034] running on GCE node
I20260924 10:01:06.636122 22210 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:01:06.637473 22210 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:01:06.638671 22210 hybrid_clock.cc:648] HybridClock initialized: now 1790244066638628 us; error 33 us; skew 500 ppm
I20260924 10:01:06.639010 22210 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:01:06 dist-test-slave-w8d2 krb5kdc[22197](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244066, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:01:06.641959 22210 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260924 10:01:06.643337 22210 webserver.cc:460] Webserver started at http://127.0.97.62:37059/ using document root <none> and password file <none>
I20260924 10:01:06.644013 22210 fs_manager.cc:359] Metadata directory not provided
I20260924 10:01:06.644110 22210 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:01:06.644348 22210 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:01:06.647132 22210 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790243735634835-388-0/minicluster-data/master-0/data/instance:
uuid: "1f1d83eace8a421aa4c0ad8e82fc31a5"
format_stamp: "Formatted at 2026-09-24 10:01:06 on dist-test-slave-w8d2"
server_key: "7a93304d0d16750c61fa2d1d32cce2b3"
server_key_iv: "fffb88d77bb8dc862030bf9008df6bfe"
server_key_version: "encryptionkey@0"
I20260924 10:01:06.647887 22210 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790243735634835-388-0/minicluster-data/master-0/wal/instance:
uuid: "1f1d83eace8a421aa4c0ad8e82fc31a5"
format_stamp: "Formatted at 2026-09-24 10:01:06 on dist-test-slave-w8d2"
server_key: "7a93304d0d16750c61fa2d1d32cce2b3"
server_key_iv: "fffb88d77bb8dc862030bf9008df6bfe"
server_key_version: "encryptionkey@0"
I20260924 10:01:06.652570 22210 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260924 10:01:06.655642 22225 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:01:06.656906 22210 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260924 10:01:06.657047 22210 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790243735634835-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790243735634835-388-0/minicluster-data/master-0/wal
uuid: "1f1d83eace8a421aa4c0ad8e82fc31a5"
format_stamp: "Formatted at 2026-09-24 10:01:06 on dist-test-slave-w8d2"
server_key: "7a93304d0d16750c61fa2d1d32cce2b3"
server_key_iv: "fffb88d77bb8dc862030bf9008df6bfe"
server_key_version: "encryptionkey@0"
I20260924 10:01:06.657214 22210 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790243735634835-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790243735634835-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790243735634835-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:01:06.673260 22210 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:01:06.677275 22210 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:01:06.677517 22210 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:01:06.694495 22210 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:35835
I20260924 10:01:06.694517 22276 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:35835 every 8 connection(s)
I20260924 10:01:06.695848 22210 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790243735634835-388-0/minicluster-data/master-0/data/info.pb
I20260924 10:01:06.699453 22277 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:01:06.703594 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu as pid 22210
I20260924 10:01:06.703779 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790243735634835-388-0/minicluster-data/master-0/wal/instance
I20260924 10:01:06.704315 388 external_mini_cluster.cc:1428] Setting key 50b91a67273c5f264bd0073718e6c899
I20260924 10:01:06.710251 22277 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1f1d83eace8a421aa4c0ad8e82fc31a5: Bootstrap starting.
I20260924 10:01:06.713647 22277 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1f1d83eace8a421aa4c0ad8e82fc31a5: Neither blocks nor log segments found. Creating new log.
I20260924 10:01:06.714821 22277 log.cc:824] T 00000000000000000000000000000000 P 1f1d83eace8a421aa4c0ad8e82fc31a5: Log is configured to *not* fsync() on all Append() calls
I20260924 10:01:06.717656 22280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:06.705993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40747 (local address 127.0.97.62:35835)
0924 10:01:06.706688 (+ 695us) server_negotiation.cc:206] Beginning negotiation
0924 10:01:06.706702 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 10:01:06.706750 (+ 48us) server_negotiation.cc:407] Connection header received
0924 10:01:06.708831 (+ 2081us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:01:06.708854 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:01:06.709249 (+ 395us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:01:06.709554 (+ 305us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:01:06.710108 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:06.711461 (+ 1353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:06.713260 (+ 1799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:06.713995 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:06.714131 (+ 136us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:01:06.715309 (+ 1178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:01:06.715318 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:01:06.715334 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:01:06.716089 (+ 755us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:01:06.716175 (+ 86us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:01:06.716213 (+ 38us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:01:06.716328 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0924 10:01:06.716570 (+ 242us) server_negotiation.cc:299] Negotiation successful
0924 10:01:06.716733 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":495,"thread_start_us":323,"threads_started":1}
I20260924 10:01:06.717808 22277 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1f1d83eace8a421aa4c0ad8e82fc31a5: No bootstrap required, opened a new log
I20260924 10:01:06.721153 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 10:01:06.727059 22277 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1f1d83eace8a421aa4c0ad8e82fc31a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f1d83eace8a421aa4c0ad8e82fc31a5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35835 } }
I20260924 10:01:06.727568 22277 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1f1d83eace8a421aa4c0ad8e82fc31a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:01:06.727691 22277 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1f1d83eace8a421aa4c0ad8e82fc31a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f1d83eace8a421aa4c0ad8e82fc31a5, State: Initialized, Role: FOLLOWER
I20260924 10:01:06.728259 22277 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1f1d83eace8a421aa4c0ad8e82fc31a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f1d83eace8a421aa4c0ad8e82fc31a5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35835 } }
I20260924 10:01:06.728470 22277 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1f1d83eace8a421aa4c0ad8e82fc31a5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:01:06.728598 22277 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1f1d83eace8a421aa4c0ad8e82fc31a5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:01:06.728740 22277 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1f1d83eace8a421aa4c0ad8e82fc31a5 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:01:06.730891 22277 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1f1d83eace8a421aa4c0ad8e82fc31a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f1d83eace8a421aa4c0ad8e82fc31a5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35835 } }
I20260924 10:01:06.731467 22277 leader_election.cc:302] T 00000000000000000000000000000000 P 1f1d83eace8a421aa4c0ad8e82fc31a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1f1d83eace8a421aa4c0ad8e82fc31a5; no voters:
I20260924 10:01:06.732136 22277 leader_election.cc:288] T 00000000000000000000000000000000 P 1f1d83eace8a421aa4c0ad8e82fc31a5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:01:06.732241 22280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:06.726197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41135 (local address 127.0.97.62:35835)
0924 10:01:06.726333 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0924 10:01:06.726337 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:01:06.726689 (+ 352us) server_negotiation.cc:407] Connection header received
0924 10:01:06.726875 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:01:06.726879 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:01:06.726937 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:01:06.727010 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:01:06.727647 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:06.728516 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:06.729805 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:06.730282 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:06.730386 (+ 104us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:01:06.731469 (+ 1083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:01:06.731473 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:01:06.731476 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:01:06.731493 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:01:06.731548 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:01:06.731563 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:01:06.731654 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0924 10:01:06.731854 (+ 200us) server_negotiation.cc:299] Negotiation successful
0924 10:01:06.731910 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260924 10:01:06.732402 22287 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1f1d83eace8a421aa4c0ad8e82fc31a5 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:01:06.733772 22287 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1f1d83eace8a421aa4c0ad8e82fc31a5 [term 1 LEADER]: Becoming Leader. State: Replica: 1f1d83eace8a421aa4c0ad8e82fc31a5, State: Running, Role: LEADER
I20260924 10:01:06.734385 22277 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1f1d83eace8a421aa4c0ad8e82fc31a5 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:01:06.734355 22287 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1f1d83eace8a421aa4c0ad8e82fc31a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f1d83eace8a421aa4c0ad8e82fc31a5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35835 } }
I20260924 10:01:06.737284 22288 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f1d83eace8a421aa4c0ad8e82fc31a5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1f1d83eace8a421aa4c0ad8e82fc31a5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f1d83eace8a421aa4c0ad8e82fc31a5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35835 } } }
I20260924 10:01:06.737673 22288 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f1d83eace8a421aa4c0ad8e82fc31a5 [sys.catalog]: This master's current role is: LEADER
I20260924 10:01:06.737567 22289 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f1d83eace8a421aa4c0ad8e82fc31a5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1f1d83eace8a421aa4c0ad8e82fc31a5. Latest consensus state: current_term: 1 leader_uuid: "1f1d83eace8a421aa4c0ad8e82fc31a5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f1d83eace8a421aa4c0ad8e82fc31a5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35835 } } }
I20260924 10:01:06.737866 22289 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f1d83eace8a421aa4c0ad8e82fc31a5 [sys.catalog]: This master's current role is: LEADER
I20260924 10:01:06.741389 22295 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:01:06.747684 22295 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:01:06.755901 22295 catalog_manager.cc:1347] Generated new cluster ID: e5dc3963b1584d648aeec8744b6b205c
I20260924 10:01:06.756028 22295 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:01:06.772424 22295 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:01:06.773694 22295 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:01:06.783002 22295 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1f1d83eace8a421aa4c0ad8e82fc31a5: Generated new TSK 0
I20260924 10:01:06.783843 22295 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 10:01:06.791944 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu pid 22210
W20260924 10:01:06.792989 22272 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:40747
W20260924 10:01:06.793623 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu pid 22210: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 10:01:06.793738 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu with pid 22210
W20260924 10:01:06.807324 22195 connection.cc:535] client connection to 127.0.97.62:35835 recv error: Network error: failed to read from TLS socket (remote: 127.0.97.62:35835): Connection reset by peer (error 104)
2026-09-24T10:01:06Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1739 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790243735634835-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:01:06 dist-test-slave-w8d2 krb5kdc[22308](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:01:06 dist-test-slave-w8d2 krb5kdc[22308](info): set up 2 sockets
Sep 24 10:01:06 dist-test-slave-w8d2 krb5kdc[22308](info): commencing operation
krb5kdc: starting...
W20260924 10:01:07.799727 388 mini_kdc.cc:121] Time spent starting KDC: real 0.974s user 0.007s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790243735634835-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790243735634835-388-0/krb5kdc/test-user.keytab.
Sep 24 10:01:07 dist-test-slave-w8d2 krb5kdc[22308](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244067, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:01:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:01:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790243735634835-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790243735634835-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260924 10:01:08.016376 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
/tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790243735634835-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790243735634835-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790243735634835-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790243735634835-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38363
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790243735634835-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:38363
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790243735634835-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790243735634835-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790243735634835-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:01:08.326225 22321 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:01:08.326531 22321 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:01:08.326651 22321 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:01:08.333873 22321 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:01:08.333998 22321 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:01:08.334069 22321 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:01:08.334148 22321 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:01:08.334203 22321 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:01:08.344086 22321 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790243735634835-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790243735634835-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:38363
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790243735634835-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38363
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790243735634835-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790243735634835-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:01:08.345332 22321 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:01:08.346587 22321 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:01:08.357394 22326 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:01:08.357415 22327 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:01:08.357470 22329 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:01:08.358028 22321 server_base.cc:1034] running on GCE node
I20260924 10:01:08.358757 22321 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:01:08.360121 22321 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:01:08.361308 22321 hybrid_clock.cc:648] HybridClock initialized: now 1790244068361250 us; error 50 us; skew 500 ppm
I20260924 10:01:08.361619 22321 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:01:08 dist-test-slave-w8d2 krb5kdc[22308](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244068, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:01:08.367368 22321 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260924 10:01:08.368765 22321 webserver.cc:460] Webserver started at http://127.0.97.62:40287/ using document root <none> and password file <none>
I20260924 10:01:08.369364 22321 fs_manager.cc:359] Metadata directory not provided
I20260924 10:01:08.369469 22321 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:01:08.369709 22321 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:01:08.372545 22321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790243735634835-388-0/minicluster-data/master-0/data/instance:
uuid: "16428cb5f4a44c80a28be69ac75fbf39"
format_stamp: "Formatted at 2026-09-24 10:01:08 on dist-test-slave-w8d2"
server_key: "4bd9f75d1959226724a59a9ca1874ee8"
server_key_iv: "9fa8e2466e8cb2c21c9bafdd79ff4833"
server_key_version: "encryptionkey@0"
I20260924 10:01:08.373222 22321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790243735634835-388-0/minicluster-data/master-0/wal/instance:
uuid: "16428cb5f4a44c80a28be69ac75fbf39"
format_stamp: "Formatted at 2026-09-24 10:01:08 on dist-test-slave-w8d2"
server_key: "4bd9f75d1959226724a59a9ca1874ee8"
server_key_iv: "9fa8e2466e8cb2c21c9bafdd79ff4833"
server_key_version: "encryptionkey@0"
I20260924 10:01:08.377930 22321 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260924 10:01:08.381071 22336 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:01:08.382263 22321 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 10:01:08.382411 22321 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790243735634835-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790243735634835-388-0/minicluster-data/master-0/wal
uuid: "16428cb5f4a44c80a28be69ac75fbf39"
format_stamp: "Formatted at 2026-09-24 10:01:08 on dist-test-slave-w8d2"
server_key: "4bd9f75d1959226724a59a9ca1874ee8"
server_key_iv: "9fa8e2466e8cb2c21c9bafdd79ff4833"
server_key_version: "encryptionkey@0"
I20260924 10:01:08.382570 22321 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790243735634835-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790243735634835-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790243735634835-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:01:08.387848 22321 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:01:08.391492 22321 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:01:08.391741 22321 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:01:08.408068 22321 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:38363
I20260924 10:01:08.408102 22387 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:38363 every 8 connection(s)
I20260924 10:01:08.409384 22321 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790243735634835-388-0/minicluster-data/master-0/data/info.pb
I20260924 10:01:08.412904 22388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:01:08.416720 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu as pid 22321
I20260924 10:01:08.416895 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790243735634835-388-0/minicluster-data/master-0/wal/instance
I20260924 10:01:08.417394 388 external_mini_cluster.cc:1428] Setting key 61f3dd773373084d0e8fb0b68bad64c2
I20260924 10:01:08.424579 22391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:08.419051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60659 (local address 127.0.97.62:38363)
0924 10:01:08.419776 (+ 725us) server_negotiation.cc:206] Beginning negotiation
0924 10:01:08.419787 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 10:01:08.419826 (+ 39us) server_negotiation.cc:407] Connection header received
0924 10:01:08.421418 (+ 1592us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:01:08.421439 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:01:08.421793 (+ 354us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:01:08.422122 (+ 329us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:01:08.423099 (+ 977us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:01:08.423104 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:01:08.423117 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:01:08.423130 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:01:08.423184 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:01:08.423203 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:01:08.423415 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0924 10:01:08.423508 (+ 93us) server_negotiation.cc:299] Negotiation successful
0924 10:01:08.423683 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":534,"thread_start_us":330,"threads_started":1}
I20260924 10:01:08.425187 22388 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16428cb5f4a44c80a28be69ac75fbf39: Bootstrap starting.
I20260924 10:01:08.427881 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 10:01:08.428205 22388 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 16428cb5f4a44c80a28be69ac75fbf39: Neither blocks nor log segments found. Creating new log.
I20260924 10:01:08.429119 22388 log.cc:824] T 00000000000000000000000000000000 P 16428cb5f4a44c80a28be69ac75fbf39: Log is configured to *not* fsync() on all Append() calls
I20260924 10:01:08.431931 22388 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16428cb5f4a44c80a28be69ac75fbf39: No bootstrap required, opened a new log
I20260924 10:01:08.433971 22391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:08.431768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56043 (local address 127.0.97.62:38363)
0924 10:01:08.431901 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0924 10:01:08.431906 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:01:08.432313 (+ 407us) server_negotiation.cc:407] Connection header received
0924 10:01:08.432463 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:01:08.432467 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:01:08.432507 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:01:08.432581 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:01:08.433399 (+ 818us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:01:08.433407 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:01:08.433412 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:01:08.433429 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:01:08.433461 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:01:08.433474 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:01:08.433582 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0924 10:01:08.433668 (+ 86us) server_negotiation.cc:299] Negotiation successful
0924 10:01:08.433723 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260924 10:01:08.439412 22388 raft_consensus.cc:348] T 00000000000000000000000000000000 P 16428cb5f4a44c80a28be69ac75fbf39 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16428cb5f4a44c80a28be69ac75fbf39" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38363 } }
I20260924 10:01:08.439752 22388 raft_consensus.cc:374] T 00000000000000000000000000000000 P 16428cb5f4a44c80a28be69ac75fbf39 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:01:08.439845 22388 raft_consensus.cc:729] T 00000000000000000000000000000000 P 16428cb5f4a44c80a28be69ac75fbf39 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16428cb5f4a44c80a28be69ac75fbf39, State: Initialized, Role: FOLLOWER
I20260924 10:01:08.440397 22388 consensus_queue.cc:260] T 00000000000000000000000000000000 P 16428cb5f4a44c80a28be69ac75fbf39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16428cb5f4a44c80a28be69ac75fbf39" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38363 } }
I20260924 10:01:08.440600 22388 raft_consensus.cc:388] T 00000000000000000000000000000000 P 16428cb5f4a44c80a28be69ac75fbf39 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:01:08.440724 22388 raft_consensus.cc:482] T 00000000000000000000000000000000 P 16428cb5f4a44c80a28be69ac75fbf39 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:01:08.440867 22388 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 16428cb5f4a44c80a28be69ac75fbf39 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:01:08.443077 22388 raft_consensus.cc:504] T 00000000000000000000000000000000 P 16428cb5f4a44c80a28be69ac75fbf39 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16428cb5f4a44c80a28be69ac75fbf39" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38363 } }
I20260924 10:01:08.443624 22388 leader_election.cc:302] T 00000000000000000000000000000000 P 16428cb5f4a44c80a28be69ac75fbf39 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 16428cb5f4a44c80a28be69ac75fbf39; no voters:
I20260924 10:01:08.444124 22388 leader_election.cc:288] T 00000000000000000000000000000000 P 16428cb5f4a44c80a28be69ac75fbf39 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:01:08.444237 22398 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 16428cb5f4a44c80a28be69ac75fbf39 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:01:08.444802 22398 raft_consensus.cc:686] T 00000000000000000000000000000000 P 16428cb5f4a44c80a28be69ac75fbf39 [term 1 LEADER]: Becoming Leader. State: Replica: 16428cb5f4a44c80a28be69ac75fbf39, State: Running, Role: LEADER
I20260924 10:01:08.445289 22398 consensus_queue.cc:237] T 00000000000000000000000000000000 P 16428cb5f4a44c80a28be69ac75fbf39 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16428cb5f4a44c80a28be69ac75fbf39" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38363 } }
I20260924 10:01:08.446007 22388 sys_catalog.cc:564] T 00000000000000000000000000000000 P 16428cb5f4a44c80a28be69ac75fbf39 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:01:08.448151 22399 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16428cb5f4a44c80a28be69ac75fbf39 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "16428cb5f4a44c80a28be69ac75fbf39" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16428cb5f4a44c80a28be69ac75fbf39" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38363 } } }
I20260924 10:01:08.448606 22399 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16428cb5f4a44c80a28be69ac75fbf39 [sys.catalog]: This master's current role is: LEADER
I20260924 10:01:08.447667 22400 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16428cb5f4a44c80a28be69ac75fbf39 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 16428cb5f4a44c80a28be69ac75fbf39. Latest consensus state: current_term: 1 leader_uuid: "16428cb5f4a44c80a28be69ac75fbf39" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16428cb5f4a44c80a28be69ac75fbf39" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38363 } } }
I20260924 10:01:08.449095 22400 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16428cb5f4a44c80a28be69ac75fbf39 [sys.catalog]: This master's current role is: LEADER
I20260924 10:01:08.450143 22404 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:01:08.454972 22404 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:01:08.463965 22404 catalog_manager.cc:1347] Generated new cluster ID: 02cad4e693864784aee79cf317ec3f50
I20260924 10:01:08.464068 22404 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:01:08.472446 22404 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:01:08.472829 22404 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:01:08.490259 22404 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 16428cb5f4a44c80a28be69ac75fbf39: Generated new TSK 0
I20260924 10:01:08.491436 22404 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 10:01:08.508728 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu pid 22321
W20260924 10:01:08.509835 22383 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:60659
W20260924 10:01:08.510473 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu pid 22321: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 10:01:08.510622 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu with pid 22321
2026-09-24T10:01:08Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1716 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790243735634835-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:01:08 dist-test-slave-w8d2 krb5kdc[22419](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:01:08 dist-test-slave-w8d2 krb5kdc[22419](info): set up 2 sockets
Sep 24 10:01:08 dist-test-slave-w8d2 krb5kdc[22419](info): commencing operation
krb5kdc: starting...
W20260924 10:01:09.519356 388 mini_kdc.cc:121] Time spent starting KDC: real 0.978s user 0.009s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790243735634835-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790243735634835-388-0/krb5kdc/test-user.keytab.
Sep 24 10:01:09 dist-test-slave-w8d2 krb5kdc[22419](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244069, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:01:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:01:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790243735634835-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790243735634835-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260924 10:01:09.742184 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
/tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790243735634835-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790243735634835-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790243735634835-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790243735634835-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35973
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790243735634835-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:35973
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790243735634835-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790243735634835-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790243735634835-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:01:10.051537 22432 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:01:10.051834 22432 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:01:10.051971 22432 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:01:10.059072 22432 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:01:10.059183 22432 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:01:10.059283 22432 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:01:10.059360 22432 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:01:10.059422 22432 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:01:10.069034 22432 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790243735634835-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790243735634835-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:35973
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790243735634835-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35973
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790243735634835-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790243735634835-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:01:10.070286 22432 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:01:10.071796 22432 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:01:10.082943 22438 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:01:10.083001 22440 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:01:10.083024 22437 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:01:10.088606 22432 server_base.cc:1034] running on GCE node
I20260924 10:01:10.089332 22432 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:01:10.090668 22432 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:01:10.091857 22432 hybrid_clock.cc:648] HybridClock initialized: now 1790244070091801 us; error 45 us; skew 500 ppm
I20260924 10:01:10.092178 22432 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:01:10 dist-test-slave-w8d2 krb5kdc[22419](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244070, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:01:10.094971 22432 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260924 10:01:10.096405 22432 webserver.cc:460] Webserver started at http://127.0.97.62:37675/ using document root <none> and password file <none>
I20260924 10:01:10.097054 22432 fs_manager.cc:359] Metadata directory not provided
I20260924 10:01:10.097164 22432 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:01:10.097426 22432 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:01:10.100255 22432 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790243735634835-388-0/minicluster-data/master-0/data/instance:
uuid: "1d3c169529944d6d81aa188a79d30398"
format_stamp: "Formatted at 2026-09-24 10:01:10 on dist-test-slave-w8d2"
server_key: "cc32e5e0a53cd9cdbc955fe6285a2180"
server_key_iv: "e8f1fe27896acfecd6d57f0ba9883a14"
server_key_version: "encryptionkey@0"
I20260924 10:01:10.100957 22432 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790243735634835-388-0/minicluster-data/master-0/wal/instance:
uuid: "1d3c169529944d6d81aa188a79d30398"
format_stamp: "Formatted at 2026-09-24 10:01:10 on dist-test-slave-w8d2"
server_key: "cc32e5e0a53cd9cdbc955fe6285a2180"
server_key_iv: "e8f1fe27896acfecd6d57f0ba9883a14"
server_key_version: "encryptionkey@0"
I20260924 10:01:10.105671 22432 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 10:01:10.108834 22447 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:01:10.110029 22432 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 10:01:10.110183 22432 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790243735634835-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790243735634835-388-0/minicluster-data/master-0/wal
uuid: "1d3c169529944d6d81aa188a79d30398"
format_stamp: "Formatted at 2026-09-24 10:01:10 on dist-test-slave-w8d2"
server_key: "cc32e5e0a53cd9cdbc955fe6285a2180"
server_key_iv: "e8f1fe27896acfecd6d57f0ba9883a14"
server_key_version: "encryptionkey@0"
I20260924 10:01:10.110349 22432 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790243735634835-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790243735634835-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790243735634835-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:01:10.126133 22432 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:01:10.130119 22432 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:01:10.130369 22432 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:01:10.146906 22432 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:35973
I20260924 10:01:10.146929 22498 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:35973 every 8 connection(s)
I20260924 10:01:10.148221 22432 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790243735634835-388-0/minicluster-data/master-0/data/info.pb
I20260924 10:01:10.151903 22499 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:01:10.152141 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu as pid 22432
I20260924 10:01:10.152307 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790243735634835-388-0/minicluster-data/master-0/wal/instance
I20260924 10:01:10.152801 388 external_mini_cluster.cc:1428] Setting key e618cfca8f16f3e796bf75cc02700baa
I20260924 10:01:10.166162 22499 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d3c169529944d6d81aa188a79d30398: Bootstrap starting.
Sep 24 10:01:10 dist-test-slave-w8d2 krb5kdc[22419](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244069, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 10:01:10.169572 22499 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1d3c169529944d6d81aa188a79d30398: Neither blocks nor log segments found. Creating new log.
I20260924 10:01:10.170950 22499 log.cc:824] T 00000000000000000000000000000000 P 1d3c169529944d6d81aa188a79d30398: Log is configured to *not* fsync() on all Append() calls
I20260924 10:01:10.174365 22499 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d3c169529944d6d81aa188a79d30398: No bootstrap required, opened a new log
I20260924 10:01:10.177390 22502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:10.154589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45163 (local address 127.0.97.62:35973)
0924 10:01:10.155827 (+ 1238us) server_negotiation.cc:206] Beginning negotiation
0924 10:01:10.155845 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0924 10:01:10.156341 (+ 496us) server_negotiation.cc:407] Connection header received
0924 10:01:10.158459 (+ 2118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:01:10.158492 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:01:10.159037 (+ 545us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:01:10.159511 (+ 474us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:01:10.160693 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:10.162607 (+ 1914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:10.164029 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:10.165058 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:10.165249 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:01:10.169478 (+ 4229us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:01:10.169490 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:01:10.169510 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:01:10.169568 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:01:10.173282 (+ 3714us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:01:10.173982 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:01:10.173991 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:01:10.174000 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:01:10.174204 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:01:10.174561 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:01:10.174565 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:01:10.174567 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:01:10.175123 (+ 556us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:01:10.175451 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0924 10:01:10.175889 (+ 438us) server_negotiation.cc:299] Negotiation successful
0924 10:01:10.176332 (+ 443us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":961,"thread_start_us":708,"threads_started":1}
I20260924 10:01:10.181319 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 10:01:10.182935 22499 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1d3c169529944d6d81aa188a79d30398 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d3c169529944d6d81aa188a79d30398" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35973 } }
I20260924 10:01:10.183357 22499 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1d3c169529944d6d81aa188a79d30398 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:01:10.183452 22499 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1d3c169529944d6d81aa188a79d30398 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d3c169529944d6d81aa188a79d30398, State: Initialized, Role: FOLLOWER
I20260924 10:01:10.184078 22499 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1d3c169529944d6d81aa188a79d30398 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d3c169529944d6d81aa188a79d30398" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35973 } }
I20260924 10:01:10.184336 22499 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1d3c169529944d6d81aa188a79d30398 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:01:10.184489 22499 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1d3c169529944d6d81aa188a79d30398 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:01:10.184645 22499 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1d3c169529944d6d81aa188a79d30398 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:01:10.187139 22499 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1d3c169529944d6d81aa188a79d30398 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d3c169529944d6d81aa188a79d30398" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35973 } }
I20260924 10:01:10.187633 22499 leader_election.cc:302] T 00000000000000000000000000000000 P 1d3c169529944d6d81aa188a79d30398 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1d3c169529944d6d81aa188a79d30398; no voters:
I20260924 10:01:10.188235 22499 leader_election.cc:288] T 00000000000000000000000000000000 P 1d3c169529944d6d81aa188a79d30398 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:01:10.188340 22509 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1d3c169529944d6d81aa188a79d30398 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:01:10.190282 22509 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1d3c169529944d6d81aa188a79d30398 [term 1 LEADER]: Becoming Leader. State: Replica: 1d3c169529944d6d81aa188a79d30398, State: Running, Role: LEADER
I20260924 10:01:10.190577 22499 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1d3c169529944d6d81aa188a79d30398 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:01:10.190858 22509 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1d3c169529944d6d81aa188a79d30398 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d3c169529944d6d81aa188a79d30398" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35973 } }
I20260924 10:01:10.194693 22510 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d3c169529944d6d81aa188a79d30398 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1d3c169529944d6d81aa188a79d30398" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d3c169529944d6d81aa188a79d30398" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35973 } } }
I20260924 10:01:10.195079 22510 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d3c169529944d6d81aa188a79d30398 [sys.catalog]: This master's current role is: LEADER
I20260924 10:01:10.198045 22511 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d3c169529944d6d81aa188a79d30398 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1d3c169529944d6d81aa188a79d30398. Latest consensus state: current_term: 1 leader_uuid: "1d3c169529944d6d81aa188a79d30398" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d3c169529944d6d81aa188a79d30398" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35973 } } }
I20260924 10:01:10.198359 22511 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d3c169529944d6d81aa188a79d30398 [sys.catalog]: This master's current role is: LEADER
I20260924 10:01:10.199168 22518 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:01:10.200723 22502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:10.185465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35399 (local address 127.0.97.62:35973)
0924 10:01:10.185627 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0924 10:01:10.185631 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:01:10.186074 (+ 443us) server_negotiation.cc:407] Connection header received
0924 10:01:10.186234 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:01:10.186240 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:01:10.186310 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:01:10.186391 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:01:10.187762 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:10.188594 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:10.190102 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:10.190703 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:10.190833 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:01:10.193639 (+ 2806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:01:10.193646 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:01:10.193650 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:01:10.193688 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:01:10.196918 (+ 3230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:01:10.197690 (+ 772us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:01:10.197695 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:01:10.197699 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:01:10.197861 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:01:10.198351 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:01:10.198359 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:01:10.198362 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:01:10.198784 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:01:10.199048 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0924 10:01:10.200144 (+ 1096us) server_negotiation.cc:299] Negotiation successful
0924 10:01:10.200385 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 10:01:10.204777 22518 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:01:10.213706 22518 catalog_manager.cc:1347] Generated new cluster ID: c0353bdae9e54404959d51e602b8a12e
I20260924 10:01:10.213828 22518 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:01:10.244321 22518 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:01:10.246215 22518 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:01:10.277128 22518 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1d3c169529944d6d81aa188a79d30398: Generated new TSK 0
I20260924 10:01:10.277997 22518 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 10:01:10.287762 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu pid 22432
I20260924 10:01:10.289036 22494 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:01:10.384372 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu with pid 22432
2026-09-24T10:01:10Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1874 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790243735634835-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:01:10 dist-test-slave-w8d2 krb5kdc[22531](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:01:10 dist-test-slave-w8d2 krb5kdc[22531](info): set up 2 sockets
Sep 24 10:01:10 dist-test-slave-w8d2 krb5kdc[22531](info): commencing operation
krb5kdc: starting...
W20260924 10:01:11.395454 388 mini_kdc.cc:121] Time spent starting KDC: real 0.979s user 0.002s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790243735634835-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790243735634835-388-0/krb5kdc/test-user.keytab.
Sep 24 10:01:11 dist-test-slave-w8d2 krb5kdc[22531](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244071, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:01:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:01:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790243735634835-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790243735634835-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260924 10:01:11.622967 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
/tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790243735634835-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790243735634835-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790243735634835-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790243735634835-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45499
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790243735634835-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:32953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45499
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790243735634835-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790243735634835-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790243735634835-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:01:11.934202 22544 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:01:11.934478 22544 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:01:11.934612 22544 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:01:11.941725 22544 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:01:11.941835 22544 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:01:11.941892 22544 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:01:11.941973 22544 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:01:11.942034 22544 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:01:11.951826 22544 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:32953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790243735634835-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790243735634835-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45499
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790243735634835-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45499
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790243735634835-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790243735634835-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:01:11.953080 22544 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:01:11.954334 22544 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:01:11.965528 22549 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:01:11.965493 22550 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:01:11.965577 22552 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:01:11.966264 22544 server_base.cc:1034] running on GCE node
I20260924 10:01:11.967077 22544 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:01:11.968537 22544 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:01:11.969724 22544 hybrid_clock.cc:648] HybridClock initialized: now 1790244071969681 us; error 32 us; skew 500 ppm
I20260924 10:01:11.970046 22544 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:01:11 dist-test-slave-w8d2 krb5kdc[22531](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244071, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:01:11.973065 22544 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260924 10:01:11.974464 22544 webserver.cc:460] Webserver started at http://127.0.97.62:45629/ using document root <none> and password file <none>
I20260924 10:01:11.975121 22544 fs_manager.cc:359] Metadata directory not provided
I20260924 10:01:11.975224 22544 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:01:11.975541 22544 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:01:11.978434 22544 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790243735634835-388-0/minicluster-data/master-0/data/instance:
uuid: "caae35033cef4c8896a549520b0e6812"
format_stamp: "Formatted at 2026-09-24 10:01:11 on dist-test-slave-w8d2"
server_key: "a29629495a9176889bf8448385776e24"
server_key_iv: "788901fb1b5128c9734ea4ce0bcbf542"
server_key_version: "encryptionkey@0"
I20260924 10:01:11.979156 22544 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790243735634835-388-0/minicluster-data/master-0/wal/instance:
uuid: "caae35033cef4c8896a549520b0e6812"
format_stamp: "Formatted at 2026-09-24 10:01:11 on dist-test-slave-w8d2"
server_key: "a29629495a9176889bf8448385776e24"
server_key_iv: "788901fb1b5128c9734ea4ce0bcbf542"
server_key_version: "encryptionkey@0"
I20260924 10:01:11.984050 22544 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260924 10:01:11.987349 22559 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:01:11.988692 22544 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 10:01:11.988854 22544 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790243735634835-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790243735634835-388-0/minicluster-data/master-0/wal
uuid: "caae35033cef4c8896a549520b0e6812"
format_stamp: "Formatted at 2026-09-24 10:01:11 on dist-test-slave-w8d2"
server_key: "a29629495a9176889bf8448385776e24"
server_key_iv: "788901fb1b5128c9734ea4ce0bcbf542"
server_key_version: "encryptionkey@0"
I20260924 10:01:11.989040 22544 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790243735634835-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790243735634835-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790243735634835-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:01:12.011471 22544 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:01:12.015672 22544 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:01:12.015940 22544 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:01:12.032784 22544 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:45499
I20260924 10:01:12.032806 22610 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:45499 every 8 connection(s)
I20260924 10:01:12.034119 22544 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790243735634835-388-0/minicluster-data/master-0/data/info.pb
I20260924 10:01:12.037675 22611 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:01:12.044204 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu as pid 22544
I20260924 10:01:12.044376 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790243735634835-388-0/minicluster-data/master-0/wal/instance
I20260924 10:01:12.044873 388 external_mini_cluster.cc:1428] Setting key 88bc036370bb5ca2b1d26ea9af5d440e
I20260924 10:01:12.048038 22611 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P caae35033cef4c8896a549520b0e6812: Bootstrap starting.
I20260924 10:01:12.051199 22611 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P caae35033cef4c8896a549520b0e6812: Neither blocks nor log segments found. Creating new log.
I20260924 10:01:12.052357 22611 log.cc:824] T 00000000000000000000000000000000 P caae35033cef4c8896a549520b0e6812: Log is configured to *not* fsync() on all Append() calls
I20260924 10:01:12.055390 22611 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P caae35033cef4c8896a549520b0e6812: No bootstrap required, opened a new log
I20260924 10:01:12.057590 22614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:12.046619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43795 (local address 127.0.97.62:45499)
0924 10:01:12.047351 (+ 732us) server_negotiation.cc:206] Beginning negotiation
0924 10:01:12.047362 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 10:01:12.047403 (+ 41us) server_negotiation.cc:407] Connection header received
0924 10:01:12.049042 (+ 1639us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:01:12.049065 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:01:12.049452 (+ 387us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:01:12.049780 (+ 328us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:01:12.050312 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:12.051688 (+ 1376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:12.053346 (+ 1658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:12.054080 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:12.054212 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:01:12.055348 (+ 1136us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:01:12.055355 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:01:12.055370 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:01:12.055945 (+ 575us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:01:12.056022 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:01:12.056047 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:01:12.056136 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0924 10:01:12.056342 (+ 206us) server_negotiation.cc:299] Negotiation successful
0924 10:01:12.056586 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":496,"thread_start_us":316,"threads_started":1}
I20260924 10:01:12.061331 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 10:01:12.063179 22611 raft_consensus.cc:348] T 00000000000000000000000000000000 P caae35033cef4c8896a549520b0e6812 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "caae35033cef4c8896a549520b0e6812" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45499 } }
I20260924 10:01:12.063607 22611 raft_consensus.cc:374] T 00000000000000000000000000000000 P caae35033cef4c8896a549520b0e6812 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:01:12.063695 22611 raft_consensus.cc:729] T 00000000000000000000000000000000 P caae35033cef4c8896a549520b0e6812 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: caae35033cef4c8896a549520b0e6812, State: Initialized, Role: FOLLOWER
I20260924 10:01:12.064289 22611 consensus_queue.cc:260] T 00000000000000000000000000000000 P caae35033cef4c8896a549520b0e6812 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "caae35033cef4c8896a549520b0e6812" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45499 } }
I20260924 10:01:12.064539 22611 raft_consensus.cc:388] T 00000000000000000000000000000000 P caae35033cef4c8896a549520b0e6812 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:01:12.064698 22611 raft_consensus.cc:482] T 00000000000000000000000000000000 P caae35033cef4c8896a549520b0e6812 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:01:12.064844 22611 raft_consensus.cc:3037] T 00000000000000000000000000000000 P caae35033cef4c8896a549520b0e6812 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:01:12.067615 22611 raft_consensus.cc:504] T 00000000000000000000000000000000 P caae35033cef4c8896a549520b0e6812 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "caae35033cef4c8896a549520b0e6812" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45499 } }
I20260924 10:01:12.068112 22611 leader_election.cc:302] T 00000000000000000000000000000000 P caae35033cef4c8896a549520b0e6812 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: caae35033cef4c8896a549520b0e6812; no voters:
I20260924 10:01:12.068735 22611 leader_election.cc:288] T 00000000000000000000000000000000 P caae35033cef4c8896a549520b0e6812 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:01:12.068846 22621 raft_consensus.cc:2781] T 00000000000000000000000000000000 P caae35033cef4c8896a549520b0e6812 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:01:12.071043 22611 sys_catalog.cc:564] T 00000000000000000000000000000000 P caae35033cef4c8896a549520b0e6812 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:01:12.071851 22614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:12.065660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39845 (local address 127.0.97.62:45499)
0924 10:01:12.065851 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0924 10:01:12.065857 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:01:12.066161 (+ 304us) server_negotiation.cc:407] Connection header received
0924 10:01:12.066326 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:01:12.066329 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:01:12.066392 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:01:12.066489 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:01:12.067115 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:12.067839 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:12.069210 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:12.069878 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:12.069952 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:01:12.071112 (+ 1160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:01:12.071118 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:01:12.071122 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:01:12.071143 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:01:12.071188 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:01:12.071212 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:01:12.071312 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0924 10:01:12.071419 (+ 107us) server_negotiation.cc:299] Negotiation successful
0924 10:01:12.071505 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 10:01:12.075520 22621 raft_consensus.cc:686] T 00000000000000000000000000000000 P caae35033cef4c8896a549520b0e6812 [term 1 LEADER]: Becoming Leader. State: Replica: caae35033cef4c8896a549520b0e6812, State: Running, Role: LEADER
I20260924 10:01:12.076196 22621 consensus_queue.cc:237] T 00000000000000000000000000000000 P caae35033cef4c8896a549520b0e6812 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "caae35033cef4c8896a549520b0e6812" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45499 } }
I20260924 10:01:12.079396 22622 sys_catalog.cc:455] T 00000000000000000000000000000000 P caae35033cef4c8896a549520b0e6812 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "caae35033cef4c8896a549520b0e6812" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "caae35033cef4c8896a549520b0e6812" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45499 } } }
I20260924 10:01:12.079809 22622 sys_catalog.cc:458] T 00000000000000000000000000000000 P caae35033cef4c8896a549520b0e6812 [sys.catalog]: This master's current role is: LEADER
I20260924 10:01:12.079402 22623 sys_catalog.cc:455] T 00000000000000000000000000000000 P caae35033cef4c8896a549520b0e6812 [sys.catalog]: SysCatalogTable state changed. Reason: New leader caae35033cef4c8896a549520b0e6812. Latest consensus state: current_term: 1 leader_uuid: "caae35033cef4c8896a549520b0e6812" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "caae35033cef4c8896a549520b0e6812" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45499 } } }
I20260924 10:01:12.080515 22623 sys_catalog.cc:458] T 00000000000000000000000000000000 P caae35033cef4c8896a549520b0e6812 [sys.catalog]: This master's current role is: LEADER
W20260924 10:01:12.084031 22635 catalog_manager.cc:1558] T 00000000000000000000000000000000 P caae35033cef4c8896a549520b0e6812: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 10:01:12.084168 22635 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 10:01:12.084290 22636 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:01:12.085969 22636 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:01:12.097252 22636 catalog_manager.cc:1347] Generated new cluster ID: b52d16ebf0fd47a3a3aca855b50dfd7a
I20260924 10:01:12.097383 22636 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:01:12.119870 22636 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:01:12.121189 22636 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:01:12.130574 22636 catalog_manager.cc:5868] T 00000000000000000000000000000000 P caae35033cef4c8896a549520b0e6812: Generated new TSK 0
I20260924 10:01:12.131453 22636 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 10:01:12.146668 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu pid 22544
W20260924 10:01:12.147692 22606 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:43795
W20260924 10:01:12.148366 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu pid 22544: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 10:01:12.148517 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu with pid 22544
2026-09-24T10:01:12Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1764 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790243735634835-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:01:12 dist-test-slave-w8d2 krb5kdc[22642](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:01:12 dist-test-slave-w8d2 krb5kdc[22642](info): set up 2 sockets
Sep 24 10:01:12 dist-test-slave-w8d2 krb5kdc[22642](info): commencing operation
krb5kdc: starting...
W20260924 10:01:13.159567 388 mini_kdc.cc:121] Time spent starting KDC: real 0.978s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790243735634835-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790243735634835-388-0/krb5kdc/test-user.keytab.
Sep 24 10:01:13 dist-test-slave-w8d2 krb5kdc[22642](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244073, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:01:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:01:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790243735634835-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790243735634835-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260924 10:01:13.396617 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
/tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790243735634835-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790243735634835-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790243735634835-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790243735634835-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40989
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790243735634835-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:40989
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790243735634835-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790243735634835-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790243735634835-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:01:13.711773 22655 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:01:13.712059 22655 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:01:13.712182 22655 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:01:13.719520 22655 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:01:13.719640 22655 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:01:13.719699 22655 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:01:13.719784 22655 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:01:13.719835 22655 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:01:13.729650 22655 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790243735634835-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790243735634835-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:40989
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790243735634835-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40989
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790243735634835-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790243735634835-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:01:13.730937 22655 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:01:13.732286 22655 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:01:13.743713 22660 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:01:13.743758 22663 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:01:13.743619 22661 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:01:13.744467 22655 server_base.cc:1034] running on GCE node
I20260924 10:01:13.745247 22655 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:01:13.746671 22655 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:01:13.747866 22655 hybrid_clock.cc:648] HybridClock initialized: now 1790244073747817 us; error 39 us; skew 500 ppm
I20260924 10:01:13.748198 22655 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:01:13 dist-test-slave-w8d2 krb5kdc[22642](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244073, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:01:13.756896 22655 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260924 10:01:13.758419 22655 webserver.cc:460] Webserver started at http://127.0.97.62:40715/ using document root <none> and password file <none>
I20260924 10:01:13.759089 22655 fs_manager.cc:359] Metadata directory not provided
I20260924 10:01:13.759202 22655 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:01:13.759509 22655 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:01:13.762387 22655 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790243735634835-388-0/minicluster-data/master-0/data/instance:
uuid: "6e6da65f9ae44b6a8d8b3b6cda5e8ea6"
format_stamp: "Formatted at 2026-09-24 10:01:13 on dist-test-slave-w8d2"
server_key: "dd553c7d61be82fc5fcccb073216fc34"
server_key_iv: "4a717795194a76338c1ef025b236eb97"
server_key_version: "encryptionkey@0"
I20260924 10:01:13.763106 22655 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790243735634835-388-0/minicluster-data/master-0/wal/instance:
uuid: "6e6da65f9ae44b6a8d8b3b6cda5e8ea6"
format_stamp: "Formatted at 2026-09-24 10:01:13 on dist-test-slave-w8d2"
server_key: "dd553c7d61be82fc5fcccb073216fc34"
server_key_iv: "4a717795194a76338c1ef025b236eb97"
server_key_version: "encryptionkey@0"
I20260924 10:01:13.768172 22655 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20260924 10:01:13.771395 22670 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:01:13.772601 22655 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260924 10:01:13.772753 22655 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790243735634835-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790243735634835-388-0/minicluster-data/master-0/wal
uuid: "6e6da65f9ae44b6a8d8b3b6cda5e8ea6"
format_stamp: "Formatted at 2026-09-24 10:01:13 on dist-test-slave-w8d2"
server_key: "dd553c7d61be82fc5fcccb073216fc34"
server_key_iv: "4a717795194a76338c1ef025b236eb97"
server_key_version: "encryptionkey@0"
I20260924 10:01:13.772928 22655 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790243735634835-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790243735634835-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790243735634835-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:01:13.778198 22655 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:01:13.782109 22655 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:01:13.782361 22655 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:01:13.799474 22655 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:40989
I20260924 10:01:13.799518 22721 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:40989 every 8 connection(s)
I20260924 10:01:13.800851 22655 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790243735634835-388-0/minicluster-data/master-0/data/info.pb
I20260924 10:01:13.804719 22722 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:01:13.809062 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu as pid 22655
I20260924 10:01:13.809243 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790243735634835-388-0/minicluster-data/master-0/wal/instance
I20260924 10:01:13.809746 388 external_mini_cluster.cc:1428] Setting key f77f16574b94a8d675e6e12d183cd61e
I20260924 10:01:13.815685 22722 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e6da65f9ae44b6a8d8b3b6cda5e8ea6: Bootstrap starting.
I20260924 10:01:13.817366 22725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:13.811495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46027 (local address 127.0.97.62:40989)
0924 10:01:13.812231 (+ 736us) server_negotiation.cc:206] Beginning negotiation
0924 10:01:13.812246 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 10:01:13.812289 (+ 43us) server_negotiation.cc:407] Connection header received
0924 10:01:13.814206 (+ 1917us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:01:13.814229 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:01:13.814553 (+ 324us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:01:13.814882 (+ 329us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:01:13.815927 (+ 1045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:01:13.815933 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:01:13.815946 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:01:13.815960 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:01:13.816019 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:01:13.816039 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:01:13.816216 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0924 10:01:13.816306 (+ 90us) server_negotiation.cc:299] Negotiation successful
0924 10:01:13.816464 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":536,"thread_start_us":325,"threads_started":1}
I20260924 10:01:13.818811 22722 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6e6da65f9ae44b6a8d8b3b6cda5e8ea6: Neither blocks nor log segments found. Creating new log.
I20260924 10:01:13.819882 22722 log.cc:824] T 00000000000000000000000000000000 P 6e6da65f9ae44b6a8d8b3b6cda5e8ea6: Log is configured to *not* fsync() on all Append() calls
I20260924 10:01:13.820685 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 10:01:13.822713 22722 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e6da65f9ae44b6a8d8b3b6cda5e8ea6: No bootstrap required, opened a new log
I20260924 10:01:13.827541 22725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:13.825335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52757 (local address 127.0.97.62:40989)
0924 10:01:13.825504 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0924 10:01:13.825514 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 10:01:13.825850 (+ 336us) server_negotiation.cc:407] Connection header received
0924 10:01:13.825971 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:01:13.825976 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:01:13.826028 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:01:13.826138 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:01:13.826885 (+ 747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:01:13.826891 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:01:13.826895 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:01:13.826912 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:01:13.826954 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:01:13.826973 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:01:13.827110 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0924 10:01:13.827153 (+ 43us) server_negotiation.cc:299] Negotiation successful
0924 10:01:13.827209 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 10:01:13.831290 22722 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6e6da65f9ae44b6a8d8b3b6cda5e8ea6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e6da65f9ae44b6a8d8b3b6cda5e8ea6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40989 } }
I20260924 10:01:13.831624 22722 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6e6da65f9ae44b6a8d8b3b6cda5e8ea6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:01:13.831693 22722 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6e6da65f9ae44b6a8d8b3b6cda5e8ea6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e6da65f9ae44b6a8d8b3b6cda5e8ea6, State: Initialized, Role: FOLLOWER
I20260924 10:01:13.832203 22722 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6e6da65f9ae44b6a8d8b3b6cda5e8ea6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e6da65f9ae44b6a8d8b3b6cda5e8ea6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40989 } }
I20260924 10:01:13.832404 22722 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6e6da65f9ae44b6a8d8b3b6cda5e8ea6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:01:13.832495 22722 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6e6da65f9ae44b6a8d8b3b6cda5e8ea6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:01:13.832623 22722 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6e6da65f9ae44b6a8d8b3b6cda5e8ea6 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:01:13.834793 22722 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6e6da65f9ae44b6a8d8b3b6cda5e8ea6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e6da65f9ae44b6a8d8b3b6cda5e8ea6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40989 } }
I20260924 10:01:13.835335 22722 leader_election.cc:302] T 00000000000000000000000000000000 P 6e6da65f9ae44b6a8d8b3b6cda5e8ea6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6e6da65f9ae44b6a8d8b3b6cda5e8ea6; no voters:
I20260924 10:01:13.835821 22722 leader_election.cc:288] T 00000000000000000000000000000000 P 6e6da65f9ae44b6a8d8b3b6cda5e8ea6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:01:13.835932 22732 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6e6da65f9ae44b6a8d8b3b6cda5e8ea6 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:01:13.836535 22732 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6e6da65f9ae44b6a8d8b3b6cda5e8ea6 [term 1 LEADER]: Becoming Leader. State: Replica: 6e6da65f9ae44b6a8d8b3b6cda5e8ea6, State: Running, Role: LEADER
I20260924 10:01:13.837039 22732 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6e6da65f9ae44b6a8d8b3b6cda5e8ea6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e6da65f9ae44b6a8d8b3b6cda5e8ea6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40989 } }
I20260924 10:01:13.837697 22722 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6e6da65f9ae44b6a8d8b3b6cda5e8ea6 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:01:13.839882 22734 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e6da65f9ae44b6a8d8b3b6cda5e8ea6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6e6da65f9ae44b6a8d8b3b6cda5e8ea6. Latest consensus state: current_term: 1 leader_uuid: "6e6da65f9ae44b6a8d8b3b6cda5e8ea6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e6da65f9ae44b6a8d8b3b6cda5e8ea6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40989 } } }
I20260924 10:01:13.840296 22734 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e6da65f9ae44b6a8d8b3b6cda5e8ea6 [sys.catalog]: This master's current role is: LEADER
I20260924 10:01:13.840539 22733 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e6da65f9ae44b6a8d8b3b6cda5e8ea6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6e6da65f9ae44b6a8d8b3b6cda5e8ea6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e6da65f9ae44b6a8d8b3b6cda5e8ea6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40989 } } }
I20260924 10:01:13.840833 22733 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e6da65f9ae44b6a8d8b3b6cda5e8ea6 [sys.catalog]: This master's current role is: LEADER
I20260924 10:01:13.841935 22741 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:01:13.846064 22741 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:01:13.854370 22741 catalog_manager.cc:1347] Generated new cluster ID: 678c005b56de4ef2a02de5680b750e92
I20260924 10:01:13.854466 22741 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:01:13.871663 22741 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:01:13.872151 22741 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:01:13.880249 22741 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6e6da65f9ae44b6a8d8b3b6cda5e8ea6: Generated new TSK 0
I20260924 10:01:13.881297 22741 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 10:01:13.894809 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu pid 22655
W20260924 10:01:13.896158 22717 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:46027
W20260924 10:01:13.896863 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu pid 22655: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 10:01:13.897009 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu with pid 22655
2026-09-24T10:01:13Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1746 ms)
[----------] 18 tests from AuthTokenIssuingTest (32206 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790243735634835-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:01:13 dist-test-slave-w8d2 krb5kdc[22753](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:01:13 dist-test-slave-w8d2 krb5kdc[22753](info): set up 2 sockets
Sep 24 10:01:13 dist-test-slave-w8d2 krb5kdc[22753](info): commencing operation
krb5kdc: starting...
W20260924 10:01:14.907631 388 mini_kdc.cc:121] Time spent starting KDC: real 0.979s user 0.002s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790243735634835-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790243735634835-388-0/krb5kdc/test-user.keytab.
Sep 24 10:01:15 dist-test-slave-w8d2 krb5kdc[22753](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244075, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:01:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:01:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790243735634835-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790243735634835-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260924 10:01:15.142560 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
/tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790243735634835-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790243735634835-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790243735634835-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790243735634835-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41177
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790243735634835-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:41177,127.0.97.61:45793,127.0.97.60:41439
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790243735634835-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790243735634835-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790243735634835-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:01:15.457835 22766 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 10:01:15.458148 22766 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:01:15.458248 22766 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:01:15.458352 22766 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:01:15.465598 22766 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:01:15.465688 22766 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:01:15.465747 22766 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:01:15.465852 22766 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:01:15.465932 22766 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:01:15.475641 22766 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790243735634835-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790243735634835-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:41177,127.0.97.61:45793,127.0.97.60:41439
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790243735634835-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41177
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790243735634835-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790243735634835-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:01:15.476920 22766 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:01:15.478209 22766 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:01:15.489048 22772 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:01:15.489076 22771 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:01:15.489460 22774 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:01:15.489970 22766 server_base.cc:1034] running on GCE node
I20260924 10:01:15.490718 22766 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:01:15.492184 22766 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:01:15.493424 22766 hybrid_clock.cc:648] HybridClock initialized: now 1790244075493347 us; error 40 us; skew 500 ppm
I20260924 10:01:15.493736 22766 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:01:15 dist-test-slave-w8d2 krb5kdc[22753](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244075, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:01:15.496662 22766 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260924 10:01:15.498023 22766 webserver.cc:460] Webserver started at http://127.0.97.62:35923/ using document root <none> and password file <none>
I20260924 10:01:15.498672 22766 fs_manager.cc:359] Metadata directory not provided
I20260924 10:01:15.498754 22766 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:01:15.499008 22766 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:01:15.502131 22766 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790243735634835-388-0/minicluster-data/master-0/data/instance:
uuid: "60762ee24c654fd99f78921bb0ddec54"
format_stamp: "Formatted at 2026-09-24 10:01:15 on dist-test-slave-w8d2"
server_key: "8d68ba0666f1b113f0f86ef36e1ba138"
server_key_iv: "8b320c904a9ed4183179e422770282ef"
server_key_version: "encryptionkey@0"
I20260924 10:01:15.502902 22766 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790243735634835-388-0/minicluster-data/master-0/wal/instance:
uuid: "60762ee24c654fd99f78921bb0ddec54"
format_stamp: "Formatted at 2026-09-24 10:01:15 on dist-test-slave-w8d2"
server_key: "8d68ba0666f1b113f0f86ef36e1ba138"
server_key_iv: "8b320c904a9ed4183179e422770282ef"
server_key_version: "encryptionkey@0"
I20260924 10:01:15.508255 22766 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260924 10:01:15.511665 22781 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:01:15.512931 22766 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 10:01:15.513064 22766 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790243735634835-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790243735634835-388-0/minicluster-data/master-0/wal
uuid: "60762ee24c654fd99f78921bb0ddec54"
format_stamp: "Formatted at 2026-09-24 10:01:15 on dist-test-slave-w8d2"
server_key: "8d68ba0666f1b113f0f86ef36e1ba138"
server_key_iv: "8b320c904a9ed4183179e422770282ef"
server_key_version: "encryptionkey@0"
I20260924 10:01:15.513238 22766 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790243735634835-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790243735634835-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790243735634835-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:01:15.540563 22766 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:01:15.544528 22766 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:01:15.544747 22766 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:01:15.561445 22766 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:41177
I20260924 10:01:15.561465 22832 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:41177 every 8 connection(s)
I20260924 10:01:15.562695 22766 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790243735634835-388-0/minicluster-data/master-0/data/info.pb
I20260924 10:01:15.565090 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu as pid 22766
I20260924 10:01:15.565316 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790243735634835-388-0/minicluster-data/master-0/wal/instance
I20260924 10:01:15.566071 388 external_mini_cluster.cc:1428] Setting key a742902c4cdb9b39dad244d944318b12
I20260924 10:01:15.566866 22833 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:01:15.575327 22833 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41177 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 10:01:15 dist-test-slave-w8d2 krb5kdc[22753](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244075, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.61@KRBTEST.COM" created.
I20260924 10:01:15.598286 22837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:15.579090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41177 (local address 127.0.0.1:56960)
0924 10:01:15.580068 (+ 978us) negotiation.cc:107] Waiting for socket to connect
0924 10:01:15.580116 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0924 10:01:15.582286 (+ 2170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:01:15.583131 (+ 845us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:01:15.583137 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:01:15.583867 (+ 730us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:01:15.584287 (+ 420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:01:15.584312 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:01:15.585824 (+ 1512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:01:15.585828 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:01:15.587438 (+ 1610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:01:15.587452 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:01:15.588343 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:01:15.588350 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:01:15.588530 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:01:15.589167 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:01:15.589197 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:01:15.591706 (+ 2509us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:01:15.594751 (+ 3045us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:01:15.594761 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:01:15.594769 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:01:15.595136 (+ 367us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:01:15.595572 (+ 436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:01:15.595575 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:01:15.595578 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:01:15.595714 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:01:15.596523 (+ 809us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:01:15.596528 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:01:15.596720 (+ 192us) client_negotiation.cc:769] Sending connection context
0924 10:01:15.596914 (+ 194us) client_negotiation.cc:240] Negotiation successful
0924 10:01:15.597279 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":681,"thread_start_us":397,"threads_started":1}
I20260924 10:01:15.598321 22836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:15.579090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56960 (local address 127.0.97.62:41177)
0924 10:01:15.580006 (+ 916us) server_negotiation.cc:206] Beginning negotiation
0924 10:01:15.580019 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 10:01:15.580237 (+ 218us) server_negotiation.cc:407] Connection header received
0924 10:01:15.582671 (+ 2434us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:01:15.582677 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:01:15.582917 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:01:15.583071 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:01:15.584491 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:15.585675 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:15.587561 (+ 1886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:15.588095 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:15.588261 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:01:15.591917 (+ 3656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:01:15.591925 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:01:15.591937 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:01:15.591965 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:01:15.594634 (+ 2669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:01:15.595288 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:01:15.595297 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:01:15.595300 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:01:15.595473 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:01:15.595806 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:01:15.595810 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:01:15.595814 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:01:15.596413 (+ 599us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:01:15.596599 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0924 10:01:15.597087 (+ 488us) server_negotiation.cc:299] Negotiation successful
0924 10:01:15.597431 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":89,"server-negotiator.queue_time_us":712,"thread_start_us":1032,"threads_started":1}
I20260924 10:01:15.601085 22833 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45793 } has no permanent_uuid. Determining permanent_uuid...
I20260924 10:01:15.601986 22837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:15.601641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:45793 (local address 127.0.0.1:46446)
0924 10:01:15.601792 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0924 10:01:15.601888 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:45793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260924 10:01:15.602198 22783 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.61:45793: connect: Connection refused (error 111)
I20260924 10:01:15.603586 22837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:15.603276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:45793 (local address 127.0.0.1:46456)
0924 10:01:15.603397 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0924 10:01:15.603468 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:45793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260924 10:01:15.603780 22833 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:45793: Network error: Client connection negotiation failed: client connection to 127.0.97.61:45793: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790243735634835-388-0/krb5kdc/kudu_127.0.97.61.keytab.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790243735634835-388-0/krb5kdc/kudu_127.0.97.61.keytab.
I20260924 10:01:15.622045 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
/tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790243735634835-388-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790243735634835-388-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790243735634835-388-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790243735634835-388-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:45793
--webserver_interface=127.0.97.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790243735634835-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:41177,127.0.97.61:45793,127.0.97.60:41439
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790243735634835-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790243735634835-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790243735634835-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 10:01:15.653088 22833 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45793 } attempt: 1
I20260924 10:01:15.654170 22837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:15.653767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:45793 (local address 127.0.0.1:46470)
0924 10:01:15.653941 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0924 10:01:15.654029 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:45793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260924 10:01:15.655093 22837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:15.654783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:45793 (local address 127.0.0.1:46484)
0924 10:01:15.654906 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0924 10:01:15.654971 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:45793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260924 10:01:15.655282 22833 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:45793: Network error: Client connection negotiation failed: client connection to 127.0.97.61:45793: connect: Connection refused (error 111)
I20260924 10:01:15.723506 22833 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45793 } attempt: 2
I20260924 10:01:15.724615 22837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:15.724148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:45793 (local address 127.0.0.1:46498)
0924 10:01:15.724388 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0924 10:01:15.724473 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:45793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260924 10:01:15.725615 22837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:15.725285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:45793 (local address 127.0.0.1:46514)
0924 10:01:15.725410 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0924 10:01:15.725486 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:45793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260924 10:01:15.725808 22833 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:45793: Network error: Client connection negotiation failed: client connection to 127.0.97.61:45793: connect: Connection refused (error 111)
I20260924 10:01:15.817011 22833 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45793 } attempt: 3
I20260924 10:01:15.818161 22837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:15.817725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:45793 (local address 127.0.0.1:46526)
0924 10:01:15.817916 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0924 10:01:15.818005 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:45793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260924 10:01:15.819164 22837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:15.818815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:45793 (local address 127.0.0.1:46534)
0924 10:01:15.818950 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0924 10:01:15.819018 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:45793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260924 10:01:15.819347 22833 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:45793: Network error: Client connection negotiation failed: client connection to 127.0.97.61:45793: connect: Connection refused (error 111)
W20260924 10:01:15.930830 22840 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 10:01:15.931133 22840 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:01:15.931203 22840 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:01:15.931324 22840 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:01:15.938364 22840 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:01:15.938462 22840 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:01:15.938524 22840 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:01:15.938575 22840 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:01:15.938633 22840 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:01:15.948347 22840 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790243735634835-388-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790243735634835-388-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:41177,127.0.97.61:45793,127.0.97.60:41439
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790243735634835-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:45793
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790243735634835-388-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.97.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790243735634835-388-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:01:15.949563 22840 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:01:15.950793 22840 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:01:15.961774 22846 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:01:15.961861 22848 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:01:15.961884 22845 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:01:15.962291 22840 server_base.cc:1034] running on GCE node
I20260924 10:01:15.962591 22833 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45793 } attempt: 4
I20260924 10:01:15.963275 22840 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:01:15.963725 22837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:15.963139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:45793 (local address 127.0.0.1:46542)
0924 10:01:15.963465 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0924 10:01:15.963560 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:45793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151}
I20260924 10:01:15.964695 22837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:15.964287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:45793 (local address 127.0.0.1:46554)
0924 10:01:15.964439 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0924 10:01:15.964548 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:45793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260924 10:01:15.964936 22840 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
W20260924 10:01:15.965018 22833 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:45793: Network error: Client connection negotiation failed: client connection to 127.0.97.61:45793: connect: Connection refused (error 111)
I20260924 10:01:15.966157 22840 hybrid_clock.cc:648] HybridClock initialized: now 1790244075966086 us; error 45 us; skew 500 ppm
I20260924 10:01:15.966472 22840 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:01:15 dist-test-slave-w8d2 krb5kdc[22753](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244075, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:01:15.969492 22840 init.cc:377] Logged in from keytab as kudu/127.0.97.61@KRBTEST.COM (short username kudu)
I20260924 10:01:15.970803 22840 webserver.cc:460] Webserver started at http://127.0.97.61:45949/ using document root <none> and password file <none>
I20260924 10:01:15.971508 22840 fs_manager.cc:359] Metadata directory not provided
I20260924 10:01:15.971606 22840 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:01:15.971894 22840 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:01:15.974772 22840 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790243735634835-388-0/minicluster-data/master-1/data/instance:
uuid: "bbf7177252cd496d8b9ad9b7ada894e8"
format_stamp: "Formatted at 2026-09-24 10:01:15 on dist-test-slave-w8d2"
server_key: "5085db30c74dc0abf4419309538b35fa"
server_key_iv: "075f09dc4e296299a70429cf9ae93982"
server_key_version: "encryptionkey@0"
I20260924 10:01:15.975556 22840 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790243735634835-388-0/minicluster-data/master-1/wal/instance:
uuid: "bbf7177252cd496d8b9ad9b7ada894e8"
format_stamp: "Formatted at 2026-09-24 10:01:15 on dist-test-slave-w8d2"
server_key: "5085db30c74dc0abf4419309538b35fa"
server_key_iv: "075f09dc4e296299a70429cf9ae93982"
server_key_version: "encryptionkey@0"
I20260924 10:01:15.980306 22840 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 10:01:15.983443 22855 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:01:15.984637 22840 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260924 10:01:15.984774 22840 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790243735634835-388-0/minicluster-data/master-1/data,/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790243735634835-388-0/minicluster-data/master-1/wal
uuid: "bbf7177252cd496d8b9ad9b7ada894e8"
format_stamp: "Formatted at 2026-09-24 10:01:15 on dist-test-slave-w8d2"
server_key: "5085db30c74dc0abf4419309538b35fa"
server_key_iv: "075f09dc4e296299a70429cf9ae93982"
server_key_version: "encryptionkey@0"
I20260924 10:01:15.984951 22840 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790243735634835-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790243735634835-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790243735634835-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:01:16.003368 22840 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:01:16.007321 22840 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:01:16.007560 22840 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:01:16.023595 22840 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:45793
I20260924 10:01:16.023598 22906 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:45793 every 8 connection(s)
I20260924 10:01:16.024904 22840 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790243735634835-388-0/minicluster-data/master-1/data/info.pb
I20260924 10:01:16.028316 22907 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:01:16.033558 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu as pid 22840
I20260924 10:01:16.033720 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790243735634835-388-0/minicluster-data/master-1/wal/instance
I20260924 10:01:16.034152 388 external_mini_cluster.cc:1428] Setting key 7aaff11aed67ea81de6bb92379a11fd0
I20260924 10:01:16.034400 22907 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41177 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 10:01:16 dist-test-slave-w8d2 krb5kdc[22753](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244075, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 10:01:16.052579 22836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:16.036851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56968 (local address 127.0.97.62:41177)
0924 10:01:16.037017 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0924 10:01:16.037021 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:01:16.038185 (+ 1164us) server_negotiation.cc:407] Connection header received
0924 10:01:16.039980 (+ 1795us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:01:16.039987 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:01:16.040062 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:01:16.040157 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:01:16.041552 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:16.042239 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:16.043725 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:16.044117 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:16.044225 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:01:16.047372 (+ 3147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:01:16.047378 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:01:16.047381 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:01:16.047413 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:01:16.049818 (+ 2405us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:01:16.050472 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:01:16.050476 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:01:16.050479 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:01:16.050592 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:01:16.050965 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:01:16.050969 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:01:16.050971 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:01:16.051280 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:01:16.051421 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0924 10:01:16.052052 (+ 631us) server_negotiation.cc:299] Negotiation successful
0924 10:01:16.052278 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 10:01:16.053509 22909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:16.037315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41177 (local address 127.0.0.1:56968)
0924 10:01:16.038024 (+ 709us) negotiation.cc:107] Waiting for socket to connect
0924 10:01:16.038068 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 10:01:16.039721 (+ 1653us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:01:16.040377 (+ 656us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:01:16.040385 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:01:16.040866 (+ 481us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:01:16.041389 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:01:16.041406 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:01:16.042381 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:01:16.042385 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:01:16.043580 (+ 1195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:01:16.043592 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:01:16.044318 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:01:16.044325 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:01:16.044535 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:01:16.045126 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:01:16.045147 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:01:16.047156 (+ 2009us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:01:16.049940 (+ 2784us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:01:16.049949 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:01:16.049965 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:01:16.050357 (+ 392us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:01:16.050690 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:01:16.050695 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:01:16.050699 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:01:16.050861 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:01:16.051392 (+ 531us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:01:16.051399 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:01:16.051719 (+ 320us) client_negotiation.cc:769] Sending connection context
0924 10:01:16.051938 (+ 219us) client_negotiation.cc:240] Negotiation successful
0924 10:01:16.052397 (+ 459us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":297,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.60@KRBTEST.COM; defaulting to no policy
I20260924 10:01:16.055146 22907 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45793 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.60@KRBTEST.COM" created.
Sep 24 10:01:16 dist-test-slave-w8d2 krb5kdc[22753](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244075, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260924 10:01:16.068830 22909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:16.055715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:45793 (local address 127.0.0.1:46564)
0924 10:01:16.055921 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0924 10:01:16.055944 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 10:01:16.056024 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:01:16.056874 (+ 850us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:01:16.056879 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:01:16.057261 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:01:16.057382 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:01:16.057403 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:01:16.058649 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:01:16.058654 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:01:16.059479 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:01:16.059490 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:01:16.060229 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:01:16.060235 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:01:16.060396 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:01:16.060911 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:01:16.060926 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:01:16.062878 (+ 1952us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:01:16.066618 (+ 3740us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:01:16.066622 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:01:16.066624 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:01:16.066965 (+ 341us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:01:16.067360 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:01:16.067363 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:01:16.067365 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:01:16.067525 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:01:16.068153 (+ 628us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:01:16.068157 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:01:16.068263 (+ 106us) client_negotiation.cc:769] Sending connection context
0924 10:01:16.068367 (+ 104us) client_negotiation.cc:240] Negotiation successful
0924 10:01:16.068578 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108}
I20260924 10:01:16.069144 22911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:16.055943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46564 (local address 127.0.97.61:45793)
0924 10:01:16.056480 (+ 537us) server_negotiation.cc:206] Beginning negotiation
0924 10:01:16.056488 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 10:01:16.056517 (+ 29us) server_negotiation.cc:407] Connection header received
0924 10:01:16.056573 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:01:16.056578 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:01:16.056747 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:01:16.056869 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:01:16.057542 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:16.058511 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:16.059590 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:16.060084 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:16.060194 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:01:16.063120 (+ 2926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:01:16.063134 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:01:16.063151 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:01:16.063198 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:01:16.066482 (+ 3284us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:01:16.067086 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:01:16.067101 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:01:16.067104 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:01:16.067265 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:01:16.067618 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:01:16.067622 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:01:16.067625 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:01:16.068017 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:01:16.068218 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0924 10:01:16.068528 (+ 310us) server_negotiation.cc:299] Negotiation successful
0924 10:01:16.068751 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":401,"thread_start_us":253,"threads_started":1}
I20260924 10:01:16.071555 22907 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41439 } has no permanent_uuid. Determining permanent_uuid...
I20260924 10:01:16.072544 22909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:16.072153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41439 (local address 127.0.0.1:37626)
0924 10:01:16.072303 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0924 10:01:16.072412 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260924 10:01:16.072768 22857 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41439: connect: Connection refused (error 111)
I20260924 10:01:16.074123 22909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:16.073837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41439 (local address 127.0.0.1:37642)
0924 10:01:16.073961 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0924 10:01:16.074032 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260924 10:01:16.074318 22907 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:41439: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41439: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790243735634835-388-0/krb5kdc/kudu_127.0.97.60.keytab.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790243735634835-388-0/krb5kdc/kudu_127.0.97.60.keytab.
I20260924 10:01:16.085536 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
/tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790243735634835-388-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790243735634835-388-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790243735634835-388-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790243735634835-388-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:41439
--webserver_interface=127.0.97.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790243735634835-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:41177,127.0.97.61:45793,127.0.97.60:41439
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790243735634835-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790243735634835-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790243735634835-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 10:01:16.123606 22907 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41439 } attempt: 1
I20260924 10:01:16.124771 22909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:16.124333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41439 (local address 127.0.0.1:37648)
0924 10:01:16.124528 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0924 10:01:16.124619 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260924 10:01:16.125752 22909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:16.125407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41439 (local address 127.0.0.1:37650)
0924 10:01:16.125568 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0924 10:01:16.125646 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260924 10:01:16.125953 22907 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:41439: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41439: connect: Connection refused (error 111)
I20260924 10:01:16.194157 22907 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41439 } attempt: 2
I20260924 10:01:16.195161 22909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:16.194735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41439 (local address 127.0.0.1:37658)
0924 10:01:16.194917 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0924 10:01:16.195013 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260924 10:01:16.196146 22909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:16.195812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41439 (local address 127.0.0.1:37664)
0924 10:01:16.195957 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0924 10:01:16.196025 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260924 10:01:16.196357 22907 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:41439: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41439: connect: Connection refused (error 111)
I20260924 10:01:16.264320 22833 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45793 } attempt: 5
Sep 24 10:01:16 dist-test-slave-w8d2 krb5kdc[22753](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244075, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260924 10:01:16.278661 22911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:16.265356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46580 (local address 127.0.97.61:45793)
0924 10:01:16.265522 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0924 10:01:16.265526 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:01:16.265545 (+ 19us) server_negotiation.cc:407] Connection header received
0924 10:01:16.265648 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:01:16.265652 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:01:16.265728 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:01:16.265807 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:01:16.266879 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:16.267677 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:16.269422 (+ 1745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:16.269823 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:16.269916 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:01:16.273710 (+ 3794us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:01:16.273715 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:01:16.273722 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:01:16.273757 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:01:16.276257 (+ 2500us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:01:16.276987 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:01:16.276991 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:01:16.276993 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:01:16.277113 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:01:16.277493 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:01:16.277496 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:01:16.277498 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:01:16.277774 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:01:16.277892 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0924 10:01:16.278273 (+ 381us) server_negotiation.cc:299] Negotiation successful
0924 10:01:16.278435 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 10:01:16.278831 22837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:16.265104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:45793 (local address 127.0.0.1:46580)
0924 10:01:16.265342 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0924 10:01:16.265367 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0924 10:01:16.265516 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:01:16.265884 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:01:16.265890 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:01:16.266443 (+ 553us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:01:16.266702 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:01:16.266718 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:01:16.267830 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:01:16.267836 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:01:16.269251 (+ 1415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:01:16.269268 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:01:16.269982 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:01:16.269988 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:01:16.270195 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:01:16.270867 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:01:16.270901 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:01:16.273486 (+ 2585us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:01:16.276382 (+ 2896us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:01:16.276388 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:01:16.276391 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:01:16.276859 (+ 468us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:01:16.277207 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:01:16.277212 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:01:16.277216 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:01:16.277392 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:01:16.277896 (+ 504us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:01:16.277902 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:01:16.278068 (+ 166us) client_negotiation.cc:769] Sending connection context
0924 10:01:16.278216 (+ 148us) client_negotiation.cc:240] Negotiation successful
0924 10:01:16.278494 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":79}
I20260924 10:01:16.280066 22833 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41439 } has no permanent_uuid. Determining permanent_uuid...
I20260924 10:01:16.281091 22837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:16.280651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41439 (local address 127.0.0.1:37678)
0924 10:01:16.280819 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0924 10:01:16.280922 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260924 10:01:16.282068 22837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:16.281673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41439 (local address 127.0.0.1:37692)
0924 10:01:16.281829 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0924 10:01:16.281937 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260924 10:01:16.282296 22833 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:41439: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41439: connect: Connection refused (error 111)
I20260924 10:01:16.287575 22907 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41439 } attempt: 3
I20260924 10:01:16.288493 22909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:16.288066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41439 (local address 127.0.0.1:37694)
0924 10:01:16.288247 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0924 10:01:16.288347 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260924 10:01:16.289541 22909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:16.289127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41439 (local address 127.0.0.1:37710)
0924 10:01:16.289344 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0924 10:01:16.289413 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127}
W20260924 10:01:16.289731 22907 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:41439: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41439: connect: Connection refused (error 111)
I20260924 10:01:16.333545 22833 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41439 } attempt: 1
I20260924 10:01:16.334770 22837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:16.334211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41439 (local address 127.0.0.1:37726)
0924 10:01:16.334453 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0924 10:01:16.334565 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260924 10:01:16.335821 22837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:16.335448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41439 (local address 127.0.0.1:37740)
0924 10:01:16.335609 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0924 10:01:16.335693 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260924 10:01:16.336040 22833 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:41439: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41439: connect: Connection refused (error 111)
W20260924 10:01:16.402315 22914 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 10:01:16.402653 22914 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:01:16.402752 22914 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:01:16.402858 22914 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260924 10:01:16.404245 22833 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41439 } attempt: 2
I20260924 10:01:16.405403 22837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:16.404884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41439 (local address 127.0.0.1:37756)
0924 10:01:16.405108 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0924 10:01:16.405214 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260924 10:01:16.406381 22837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:16.406000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41439 (local address 127.0.0.1:37768)
0924 10:01:16.406170 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0924 10:01:16.406254 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260924 10:01:16.406608 22833 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:41439: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41439: connect: Connection refused (error 111)
W20260924 10:01:16.410717 22914 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:01:16.410795 22914 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:01:16.410880 22914 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:01:16.410946 22914 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:01:16.411010 22914 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:01:16.421036 22914 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790243735634835-388-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790243735634835-388-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:41177,127.0.97.61:45793,127.0.97.60:41439
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790243735634835-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:41439
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790243735634835-388-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.97.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790243735634835-388-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:01:16.422343 22914 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:01:16.423728 22914 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 10:01:16.432921 22907 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41439 } attempt: 4
I20260924 10:01:16.434034 22909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:16.433528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41439 (local address 127.0.0.1:37780)
0924 10:01:16.433735 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0924 10:01:16.433853 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260924 10:01:16.435189 22909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:16.434753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41439 (local address 127.0.0.1:37792)
0924 10:01:16.434982 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0924 10:01:16.435060 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130}
W20260924 10:01:16.435164 22920 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:01:16.435334 22922 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:01:16.435561 22907 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:41439: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41439: connect: Connection refused (error 111)
W20260924 10:01:16.435644 22919 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:01:16.436555 22914 server_base.cc:1034] running on GCE node
I20260924 10:01:16.437238 22914 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:01:16.438556 22914 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:01:16.439759 22914 hybrid_clock.cc:648] HybridClock initialized: now 1790244076439696 us; error 35 us; skew 500 ppm
I20260924 10:01:16.440055 22914 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:01:16 dist-test-slave-w8d2 krb5kdc[22753](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244076, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:01:16.442848 22914 init.cc:377] Logged in from keytab as kudu/127.0.97.60@KRBTEST.COM (short username kudu)
I20260924 10:01:16.444200 22914 webserver.cc:460] Webserver started at http://127.0.97.60:42435/ using document root <none> and password file <none>
I20260924 10:01:16.444841 22914 fs_manager.cc:359] Metadata directory not provided
I20260924 10:01:16.444934 22914 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:01:16.445214 22914 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:01:16.448071 22914 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790243735634835-388-0/minicluster-data/master-2/data/instance:
uuid: "015aa72301464c2a8c177c26410e6efd"
format_stamp: "Formatted at 2026-09-24 10:01:16 on dist-test-slave-w8d2"
server_key: "3077352ab717a990d792d41908916d2f"
server_key_iv: "badbf697672d1293bba6d5f0644bee80"
server_key_version: "encryptionkey@0"
I20260924 10:01:16.448809 22914 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790243735634835-388-0/minicluster-data/master-2/wal/instance:
uuid: "015aa72301464c2a8c177c26410e6efd"
format_stamp: "Formatted at 2026-09-24 10:01:16 on dist-test-slave-w8d2"
server_key: "3077352ab717a990d792d41908916d2f"
server_key_iv: "badbf697672d1293bba6d5f0644bee80"
server_key_version: "encryptionkey@0"
I20260924 10:01:16.453658 22914 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 10:01:16.456873 22929 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:01:16.458074 22914 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 10:01:16.458214 22914 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790243735634835-388-0/minicluster-data/master-2/data,/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790243735634835-388-0/minicluster-data/master-2/wal
uuid: "015aa72301464c2a8c177c26410e6efd"
format_stamp: "Formatted at 2026-09-24 10:01:16 on dist-test-slave-w8d2"
server_key: "3077352ab717a990d792d41908916d2f"
server_key_iv: "badbf697672d1293bba6d5f0644bee80"
server_key_version: "encryptionkey@0"
I20260924 10:01:16.458421 22914 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790243735634835-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790243735634835-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790243735634835-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:01:16.485584 22914 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:01:16.489812 22914 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:01:16.490051 22914 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:01:16.506435 22914 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:41439
I20260924 10:01:16.506455 22980 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:41439 every 8 connection(s)
I20260924 10:01:16.507860 22914 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790243735634835-388-0/minicluster-data/master-2/data/info.pb
I20260924 10:01:16.511370 22981 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:01:16.512790 22833 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41439 } attempt: 3
I20260924 10:01:16.517551 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu as pid 22914
I20260924 10:01:16.517714 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790243735634835-388-0/minicluster-data/master-2/wal/instance
I20260924 10:01:16.518133 388 external_mini_cluster.cc:1428] Setting key 1a5d1f009d3d83bafdb8fe3322bb4705
I20260924 10:01:16.520709 22981 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41177 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 10:01:16 dist-test-slave-w8d2 krb5kdc[22753](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244075, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Sep 24 10:01:16 dist-test-slave-w8d2 krb5kdc[22753](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244076, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
Sep 24 10:01:16 dist-test-slave-w8d2 krb5kdc[22753](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244075, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260924 10:01:16.536613 22837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:16.513333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41439 (local address 127.0.0.1:37796)
0924 10:01:16.513537 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0924 10:01:16.513556 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 10:01:16.513652 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:01:16.518222 (+ 4570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:01:16.518229 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:01:16.518730 (+ 501us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:01:16.518927 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:01:16.518941 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:01:16.521533 (+ 2592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:01:16.521540 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:01:16.523091 (+ 1551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:01:16.523110 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:01:16.525806 (+ 2696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:01:16.525814 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:01:16.526039 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:01:16.526679 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:01:16.526716 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:01:16.529192 (+ 2476us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:01:16.533315 (+ 4123us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:01:16.533320 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:01:16.533325 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:01:16.533806 (+ 481us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:01:16.534318 (+ 512us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:01:16.534323 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:01:16.534327 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:01:16.534562 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:01:16.535451 (+ 889us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:01:16.535456 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:01:16.535648 (+ 192us) client_negotiation.cc:769] Sending connection context
0924 10:01:16.535829 (+ 181us) client_negotiation.cc:240] Negotiation successful
0924 10:01:16.536148 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260924 10:01:16.537961 22983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:16.514287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37796 (local address 127.0.97.60:41439)
0924 10:01:16.514988 (+ 701us) server_negotiation.cc:206] Beginning negotiation
0924 10:01:16.514999 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 10:01:16.515036 (+ 37us) server_negotiation.cc:407] Connection header received
0924 10:01:16.517281 (+ 2245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:01:16.517308 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:01:16.517809 (+ 501us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:01:16.518245 (+ 436us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:01:16.519125 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:16.521290 (+ 2165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:16.523333 (+ 2043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:16.525644 (+ 2311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:16.525800 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:01:16.529429 (+ 3629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:01:16.529437 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:01:16.529448 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:01:16.529480 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:01:16.533174 (+ 3694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:01:16.533949 (+ 775us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:01:16.533962 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:01:16.533971 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:01:16.534182 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:01:16.534658 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:01:16.534663 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:01:16.534667 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:01:16.535307 (+ 640us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:01:16.535475 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0924 10:01:16.536104 (+ 629us) server_negotiation.cc:299] Negotiation successful
0924 10:01:16.536526 (+ 422us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":508,"thread_start_us":304,"threads_started":1}
I20260924 10:01:16.540584 22836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:16.524315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56978 (local address 127.0.97.62:41177)
0924 10:01:16.524515 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0924 10:01:16.524521 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:01:16.525424 (+ 903us) server_negotiation.cc:407] Connection header received
0924 10:01:16.525697 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:01:16.525703 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:01:16.525795 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:01:16.525863 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:01:16.527717 (+ 1854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:16.528583 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:16.530428 (+ 1845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:16.530837 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:16.530938 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:01:16.534610 (+ 3672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:01:16.534618 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:01:16.534623 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:01:16.534660 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:01:16.537741 (+ 3081us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:01:16.538645 (+ 904us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:01:16.538651 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:01:16.538654 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:01:16.538783 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:01:16.539137 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:01:16.539142 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:01:16.539146 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:01:16.539484 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:01:16.539611 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0924 10:01:16.540041 (+ 430us) server_negotiation.cc:299] Negotiation successful
0924 10:01:16.540232 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 10:01:16.540639 22986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:16.524232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41177 (local address 127.0.0.1:56978)
0924 10:01:16.525288 (+ 1056us) negotiation.cc:107] Waiting for socket to connect
0924 10:01:16.525337 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0924 10:01:16.525549 (+ 212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:01:16.526101 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:01:16.526109 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:01:16.527255 (+ 1146us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:01:16.527570 (+ 315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:01:16.527581 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:01:16.528771 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:01:16.528776 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:01:16.530305 (+ 1529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:01:16.530317 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:01:16.531010 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:01:16.531017 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:01:16.531197 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:01:16.531873 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:01:16.531896 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:01:16.534419 (+ 2523us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:01:16.537859 (+ 3440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:01:16.537870 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:01:16.537877 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:01:16.538504 (+ 627us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:01:16.538887 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:01:16.538892 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:01:16.538896 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:01:16.539037 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:01:16.539612 (+ 575us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:01:16.539619 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:01:16.539784 (+ 165us) client_negotiation.cc:769] Sending connection context
0924 10:01:16.539928 (+ 144us) client_negotiation.cc:240] Negotiation successful
0924 10:01:16.540250 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":841,"thread_start_us":698,"threads_started":1}
I20260924 10:01:16.541980 22981 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45793 } has no permanent_uuid. Determining permanent_uuid...
I20260924 10:01:16.545207 22833 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60762ee24c654fd99f78921bb0ddec54: Bootstrap starting.
I20260924 10:01:16.546284 22985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:16.519093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37802 (local address 127.0.97.60:41439)
0924 10:01:16.525024 (+ 5931us) server_negotiation.cc:206] Beginning negotiation
0924 10:01:16.525029 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:01:16.526538 (+ 1509us) server_negotiation.cc:407] Connection header received
0924 10:01:16.526699 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:01:16.526704 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:01:16.526778 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:01:16.526901 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:01:16.528469 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:16.529736 (+ 1267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:16.531453 (+ 1717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:16.532024 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:16.532139 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:01:16.536544 (+ 4405us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:01:16.536551 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:01:16.536555 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:01:16.536598 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:01:16.543518 (+ 6920us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:01:16.544298 (+ 780us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:01:16.544303 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:01:16.544307 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:01:16.544432 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:01:16.544820 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:01:16.544824 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:01:16.544827 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:01:16.545182 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:01:16.545435 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0924 10:01:16.545711 (+ 276us) server_negotiation.cc:299] Negotiation successful
0924 10:01:16.545910 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":53,"server-negotiator.queue_time_us":5779,"thread_start_us":5616,"threads_started":1}
Sep 24 10:01:16 dist-test-slave-w8d2 krb5kdc[22753](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244076, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260924 10:01:16.549634 22833 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 60762ee24c654fd99f78921bb0ddec54: Neither blocks nor log segments found. Creating new log.
I20260924 10:01:16.550971 22833 log.cc:824] T 00000000000000000000000000000000 P 60762ee24c654fd99f78921bb0ddec54: Log is configured to *not* fsync() on all Append() calls
I20260924 10:01:16.554168 22833 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60762ee24c654fd99f78921bb0ddec54: No bootstrap required, opened a new log
I20260924 10:01:16.555694 22986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:16.542546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:45793 (local address 127.0.0.1:46596)
0924 10:01:16.542689 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0924 10:01:16.542707 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 10:01:16.542798 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:01:16.543098 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:01:16.543101 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:01:16.543694 (+ 593us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:01:16.543835 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:01:16.543842 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:01:16.544933 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:01:16.544939 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:01:16.546266 (+ 1327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:01:16.546283 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:01:16.547407 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:01:16.547412 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:01:16.547536 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:01:16.548067 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:01:16.548088 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:01:16.549995 (+ 1907us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:01:16.553164 (+ 3169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:01:16.553168 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:01:16.553172 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:01:16.553620 (+ 448us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:01:16.554034 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:01:16.554038 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:01:16.554042 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:01:16.554206 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:01:16.554800 (+ 594us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:01:16.554805 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:01:16.554956 (+ 151us) client_negotiation.cc:769] Sending connection context
0924 10:01:16.555103 (+ 147us) client_negotiation.cc:240] Negotiation successful
0924 10:01:16.555384 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260924 10:01:16.555821 22911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:16.542657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46596 (local address 127.0.97.61:45793)
0924 10:01:16.542836 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0924 10:01:16.542842 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:01:16.542861 (+ 19us) server_negotiation.cc:407] Connection header received
0924 10:01:16.542917 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:01:16.542922 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:01:16.542995 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:01:16.543097 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:01:16.543968 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:16.544781 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:16.546432 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:16.547050 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:16.547153 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:01:16.550175 (+ 3022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:01:16.550180 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:01:16.550183 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:01:16.550214 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:01:16.553038 (+ 2824us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:01:16.553753 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:01:16.553758 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:01:16.553761 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:01:16.553916 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:01:16.554354 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:01:16.554359 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:01:16.554362 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:01:16.554668 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:01:16.554835 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0924 10:01:16.555212 (+ 377us) server_negotiation.cc:299] Negotiation successful
0924 10:01:16.555462 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
Sep 24 10:01:16 dist-test-slave-w8d2 krb5kdc[22753](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244075, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 10:01:16.556816 22981 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41439 } has no permanent_uuid. Determining permanent_uuid...
I20260924 10:01:16.563933 22836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:16.547812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56994 (local address 127.0.97.62:41177)
0924 10:01:16.547972 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0924 10:01:16.547977 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:01:16.548046 (+ 69us) server_negotiation.cc:407] Connection header received
0924 10:01:16.548202 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:01:16.548208 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:01:16.548277 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:01:16.548387 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:01:16.549790 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:16.550910 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:16.552338 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:16.552780 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:16.552881 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:01:16.557228 (+ 4347us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:01:16.557235 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:01:16.557240 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:01:16.557273 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:01:16.560736 (+ 3463us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:01:16.561603 (+ 867us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:01:16.561608 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:01:16.561611 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:01:16.561795 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:01:16.562357 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:01:16.562362 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:01:16.562366 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:01:16.562735 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:01:16.563025 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0924 10:01:16.563378 (+ 353us) server_negotiation.cc:299] Negotiation successful
0924 10:01:16.563595 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
Sep 24 10:01:16 dist-test-slave-w8d2 krb5kdc[22753](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244076, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260924 10:01:16.568059 22833 raft_consensus.cc:348] T 00000000000000000000000000000000 P 60762ee24c654fd99f78921bb0ddec54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60762ee24c654fd99f78921bb0ddec54" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41177 } } peers { permanent_uuid: "bbf7177252cd496d8b9ad9b7ada894e8" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45793 } } peers { permanent_uuid: "015aa72301464c2a8c177c26410e6efd" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41439 } }
I20260924 10:01:16.568591 22833 raft_consensus.cc:729] T 00000000000000000000000000000000 P 60762ee24c654fd99f78921bb0ddec54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60762ee24c654fd99f78921bb0ddec54, State: Initialized, Role: FOLLOWER
I20260924 10:01:16.569164 22833 consensus_queue.cc:260] T 00000000000000000000000000000000 P 60762ee24c654fd99f78921bb0ddec54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60762ee24c654fd99f78921bb0ddec54" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41177 } } peers { permanent_uuid: "bbf7177252cd496d8b9ad9b7ada894e8" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45793 } } peers { permanent_uuid: "015aa72301464c2a8c177c26410e6efd" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41439 } }
I20260924 10:01:16.570544 22986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:16.557453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41439 (local address 127.0.0.1:37814)
0924 10:01:16.557778 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0924 10:01:16.557796 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 10:01:16.557896 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:01:16.558146 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:01:16.558151 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:01:16.558497 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:01:16.558622 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:01:16.558629 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:01:16.559693 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:01:16.559699 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:01:16.560497 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:01:16.560507 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:01:16.561444 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:01:16.561450 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:01:16.561638 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:01:16.562198 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:01:16.562219 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:01:16.564621 (+ 2402us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:01:16.568253 (+ 3632us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:01:16.568258 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:01:16.568262 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:01:16.568703 (+ 441us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:01:16.569026 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:01:16.569030 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:01:16.569034 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:01:16.569144 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:01:16.569676 (+ 532us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:01:16.569680 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:01:16.569815 (+ 135us) client_negotiation.cc:769] Sending connection context
0924 10:01:16.569932 (+ 117us) client_negotiation.cc:240] Negotiation successful
0924 10:01:16.570208 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224}
I20260924 10:01:16.571811 22985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:16.557631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37814 (local address 127.0.97.60:41439)
0924 10:01:16.557799 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0924 10:01:16.557804 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:01:16.557843 (+ 39us) server_negotiation.cc:407] Connection header received
0924 10:01:16.557985 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:01:16.557990 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:01:16.558047 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:01:16.558117 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:01:16.558751 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:16.559575 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:16.560613 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:16.561169 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:16.561282 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:01:16.564795 (+ 3513us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:01:16.564800 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:01:16.564805 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:01:16.564971 (+ 166us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:01:16.568125 (+ 3154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:01:16.568818 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:01:16.568823 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:01:16.568827 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:01:16.568935 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:01:16.569229 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:01:16.569234 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:01:16.569236 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:01:16.569512 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:01:16.569636 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0924 10:01:16.571002 (+ 1366us) server_negotiation.cc:299] Negotiation successful
0924 10:01:16.571201 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
Sep 24 10:01:16 dist-test-slave-w8d2 krb5kdc[22753](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244075, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260924 10:01:16.574275 22988 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60762ee24c654fd99f78921bb0ddec54 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60762ee24c654fd99f78921bb0ddec54" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41177 } } peers { permanent_uuid: "bbf7177252cd496d8b9ad9b7ada894e8" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45793 } } peers { permanent_uuid: "015aa72301464c2a8c177c26410e6efd" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41439 } } }
I20260924 10:01:16.575788 22988 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60762ee24c654fd99f78921bb0ddec54 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 10:01:16.575933 22833 sys_catalog.cc:564] T 00000000000000000000000000000000 P 60762ee24c654fd99f78921bb0ddec54 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:01:16.577394 22981 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 015aa72301464c2a8c177c26410e6efd: Bootstrap starting.
I20260924 10:01:16.581310 22981 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 015aa72301464c2a8c177c26410e6efd: Neither blocks nor log segments found. Creating new log.
I20260924 10:01:16.582028 22911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:16.565539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46604 (local address 127.0.97.61:45793)
0924 10:01:16.565698 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0924 10:01:16.565703 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:01:16.565719 (+ 16us) server_negotiation.cc:407] Connection header received
0924 10:01:16.565880 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:01:16.565886 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:01:16.565949 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:01:16.566023 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:01:16.567112 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:16.568144 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:16.569778 (+ 1634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:16.570440 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:16.570536 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:01:16.575613 (+ 5077us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:01:16.575619 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:01:16.575624 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:01:16.575662 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:01:16.578995 (+ 3333us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:01:16.579819 (+ 824us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:01:16.579825 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:01:16.579829 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:01:16.579989 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:01:16.580494 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:01:16.580499 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:01:16.580502 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:01:16.580850 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:01:16.581098 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0924 10:01:16.581413 (+ 315us) server_negotiation.cc:299] Negotiation successful
0924 10:01:16.581658 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 10:01:16.582675 22981 log.cc:824] T 00000000000000000000000000000000 P 015aa72301464c2a8c177c26410e6efd: Log is configured to *not* fsync() on all Append() calls
I20260924 10:01:16.586232 22981 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 015aa72301464c2a8c177c26410e6efd: No bootstrap required, opened a new log
I20260924 10:01:16.590142 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260924 10:01:16.590785 22999 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 60762ee24c654fd99f78921bb0ddec54: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 10:01:16.590891 22999 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 10:01:16.593433 22808 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "60762ee24c654fd99f78921bb0ddec54"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56994
I20260924 10:01:16.593847 22808 raft_consensus.cc:482] T 00000000000000000000000000000000 P 60762ee24c654fd99f78921bb0ddec54 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 10:01:16.594027 22808 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 60762ee24c654fd99f78921bb0ddec54 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:01:16.597536 22808 raft_consensus.cc:504] T 00000000000000000000000000000000 P 60762ee24c654fd99f78921bb0ddec54 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60762ee24c654fd99f78921bb0ddec54" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41177 } } peers { permanent_uuid: "bbf7177252cd496d8b9ad9b7ada894e8" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45793 } } peers { permanent_uuid: "015aa72301464c2a8c177c26410e6efd" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41439 } }
I20260924 10:01:16.599023 22808 leader_election.cc:288] T 00000000000000000000000000000000 P 60762ee24c654fd99f78921bb0ddec54 [CANDIDATE]: Term 1 election: Requested vote from peers bbf7177252cd496d8b9ad9b7ada894e8 (127.0.97.61:45793), 015aa72301464c2a8c177c26410e6efd (127.0.97.60:41439)
I20260924 10:01:16.600617 22981 raft_consensus.cc:348] T 00000000000000000000000000000000 P 015aa72301464c2a8c177c26410e6efd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60762ee24c654fd99f78921bb0ddec54" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41177 } } peers { permanent_uuid: "bbf7177252cd496d8b9ad9b7ada894e8" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45793 } } peers { permanent_uuid: "015aa72301464c2a8c177c26410e6efd" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41439 } }
I20260924 10:01:16.601125 22981 raft_consensus.cc:729] T 00000000000000000000000000000000 P 015aa72301464c2a8c177c26410e6efd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 015aa72301464c2a8c177c26410e6efd, State: Initialized, Role: FOLLOWER
I20260924 10:01:16.601672 22981 consensus_queue.cc:260] T 00000000000000000000000000000000 P 015aa72301464c2a8c177c26410e6efd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60762ee24c654fd99f78921bb0ddec54" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41177 } } peers { permanent_uuid: "bbf7177252cd496d8b9ad9b7ada894e8" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45793 } } peers { permanent_uuid: "015aa72301464c2a8c177c26410e6efd" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41439 } }
I20260924 10:01:16.590302 22766 master_runner.cc:186] Error getting master registration for 127.0.97.61:45793: OK, instance_id { permanent_uuid: "bbf7177252cd496d8b9ad9b7ada894e8" instance_seqno: 1790244076018892 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 10:01:16.606535 22766 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 10:01:16.608384 23003 sys_catalog.cc:455] T 00000000000000000000000000000000 P 015aa72301464c2a8c177c26410e6efd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60762ee24c654fd99f78921bb0ddec54" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41177 } } peers { permanent_uuid: "bbf7177252cd496d8b9ad9b7ada894e8" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45793 } } peers { permanent_uuid: "015aa72301464c2a8c177c26410e6efd" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41439 } } }
I20260924 10:01:16.609542 23003 sys_catalog.cc:458] T 00000000000000000000000000000000 P 015aa72301464c2a8c177c26410e6efd [sys.catalog]: This master's current role is: FOLLOWER
I20260924 10:01:16.609661 22981 sys_catalog.cc:564] T 00000000000000000000000000000000 P 015aa72301464c2a8c177c26410e6efd [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:01:16.609766 22956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "60762ee24c654fd99f78921bb0ddec54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "015aa72301464c2a8c177c26410e6efd"
I20260924 10:01:16.611469 22956 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 015aa72301464c2a8c177c26410e6efd [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:01:16.615306 22956 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 015aa72301464c2a8c177c26410e6efd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60762ee24c654fd99f78921bb0ddec54 in term 1.
I20260924 10:01:16.616073 22783 leader_election.cc:302] T 00000000000000000000000000000000 P 60762ee24c654fd99f78921bb0ddec54 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 015aa72301464c2a8c177c26410e6efd, 60762ee24c654fd99f78921bb0ddec54; no voters:
I20260924 10:01:16.616843 22988 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 60762ee24c654fd99f78921bb0ddec54 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:01:16.629773 22988 raft_consensus.cc:686] T 00000000000000000000000000000000 P 60762ee24c654fd99f78921bb0ddec54 [term 1 LEADER]: Becoming Leader. State: Replica: 60762ee24c654fd99f78921bb0ddec54, State: Running, Role: LEADER
I20260924 10:01:16.630471 22988 consensus_queue.cc:237] T 00000000000000000000000000000000 P 60762ee24c654fd99f78921bb0ddec54 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60762ee24c654fd99f78921bb0ddec54" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41177 } } peers { permanent_uuid: "bbf7177252cd496d8b9ad9b7ada894e8" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45793 } } peers { permanent_uuid: "015aa72301464c2a8c177c26410e6efd" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41439 } }
I20260924 10:01:16.638692 23009 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60762ee24c654fd99f78921bb0ddec54 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 60762ee24c654fd99f78921bb0ddec54. Latest consensus state: current_term: 1 leader_uuid: "60762ee24c654fd99f78921bb0ddec54" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60762ee24c654fd99f78921bb0ddec54" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41177 } } peers { permanent_uuid: "bbf7177252cd496d8b9ad9b7ada894e8" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45793 } } peers { permanent_uuid: "015aa72301464c2a8c177c26410e6efd" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41439 } } }
I20260924 10:01:16.639171 23009 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60762ee24c654fd99f78921bb0ddec54 [sys.catalog]: This master's current role is: LEADER
I20260924 10:01:16.640609 23014 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:01:16.654253 23014 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:01:16.660594 22836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:16.618159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57008 (local address 127.0.97.62:41177)
0924 10:01:16.618342 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0924 10:01:16.618348 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:01:16.630543 (+ 12195us) server_negotiation.cc:407] Connection header received
0924 10:01:16.630768 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:01:16.630774 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:01:16.630868 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:01:16.630986 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:01:16.641166 (+ 10180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:16.642574 (+ 1408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:16.644568 (+ 1994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:16.645569 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:16.645703 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:01:16.651315 (+ 5612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:01:16.651323 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:01:16.651328 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:01:16.651380 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:01:16.655209 (+ 3829us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:01:16.656699 (+ 1490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:01:16.656706 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:01:16.656709 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:01:16.656975 (+ 266us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:01:16.657672 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:01:16.657678 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:01:16.657681 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:01:16.658315 (+ 634us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:01:16.658703 (+ 388us) server_negotiation.cc:1035] Waiting for connection context
0924 10:01:16.659664 (+ 961us) server_negotiation.cc:299] Negotiation successful
0924 10:01:16.660042 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 10:01:16.664485 22956 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 015aa72301464c2a8c177c26410e6efd [term 1 FOLLOWER]: Refusing update from remote peer 60762ee24c654fd99f78921bb0ddec54: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:01:16.665638 22985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:16.642781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37830 (local address 127.0.97.60:41439)
0924 10:01:16.642945 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0924 10:01:16.642950 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:01:16.643781 (+ 831us) server_negotiation.cc:407] Connection header received
0924 10:01:16.643936 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:01:16.643942 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:01:16.644011 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:01:16.644088 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:01:16.646070 (+ 1982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:16.646777 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:16.648578 (+ 1801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:16.649252 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:16.649359 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:01:16.652263 (+ 2904us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:01:16.652270 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:01:16.652275 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:01:16.652316 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:01:16.660270 (+ 7954us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:01:16.662185 (+ 1915us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:01:16.662191 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:01:16.662194 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:01:16.662341 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:01:16.663844 (+ 1503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:01:16.663850 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:01:16.663854 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:01:16.664263 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:01:16.664516 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0924 10:01:16.665078 (+ 562us) server_negotiation.cc:299] Negotiation successful
0924 10:01:16.665310 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 10:01:16.665789 22988 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 60762ee24c654fd99f78921bb0ddec54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "015aa72301464c2a8c177c26410e6efd" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:01:16.673239 22911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:16.631437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46620 (local address 127.0.97.61:45793)
0924 10:01:16.631630 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0924 10:01:16.631636 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:01:16.642662 (+ 11026us) server_negotiation.cc:407] Connection header received
0924 10:01:16.644805 (+ 2143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:01:16.644811 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:01:16.644891 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:01:16.644958 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:01:16.647646 (+ 2688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:16.648720 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:16.656705 (+ 7985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:16.657371 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:16.657468 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:01:16.666232 (+ 8764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:01:16.666240 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:01:16.666245 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:01:16.666299 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:01:16.669933 (+ 3634us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:01:16.670820 (+ 887us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:01:16.670826 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:01:16.670830 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:01:16.671004 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:01:16.671484 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:01:16.671489 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:01:16.671493 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:01:16.671891 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:01:16.672127 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0924 10:01:16.672656 (+ 529us) server_negotiation.cc:299] Negotiation successful
0924 10:01:16.672908 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260924 10:01:16.682263 23003 sys_catalog.cc:455] T 00000000000000000000000000000000 P 015aa72301464c2a8c177c26410e6efd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 60762ee24c654fd99f78921bb0ddec54. Latest consensus state: current_term: 1 leader_uuid: "60762ee24c654fd99f78921bb0ddec54" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60762ee24c654fd99f78921bb0ddec54" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41177 } } peers { permanent_uuid: "bbf7177252cd496d8b9ad9b7ada894e8" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45793 } } peers { permanent_uuid: "015aa72301464c2a8c177c26410e6efd" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41439 } } }
I20260924 10:01:16.682830 23003 sys_catalog.cc:458] T 00000000000000000000000000000000 P 015aa72301464c2a8c177c26410e6efd [sys.catalog]: This master's current role is: FOLLOWER
I20260924 10:01:16.684434 23016 mvcc.cc:205] Tried to move back new op lower bound from 7332839738004406272 to 7332839737889628160. Current Snapshot: MvccSnapshot[applied={T|T < 7332839738004406272}]
I20260924 10:01:16.685971 23019 mvcc.cc:205] Tried to move back new op lower bound from 7332839738004406272 to 7332839737889628160. Current Snapshot: MvccSnapshot[applied={T|T < 7332839738004406272}]
I20260924 10:01:16.686745 23009 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60762ee24c654fd99f78921bb0ddec54 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "60762ee24c654fd99f78921bb0ddec54" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60762ee24c654fd99f78921bb0ddec54" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41177 } } peers { permanent_uuid: "bbf7177252cd496d8b9ad9b7ada894e8" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45793 } } peers { permanent_uuid: "015aa72301464c2a8c177c26410e6efd" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41439 } } }
I20260924 10:01:16.687001 23009 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60762ee24c654fd99f78921bb0ddec54 [sys.catalog]: This master's current role is: LEADER
I20260924 10:01:16.691705 23014 catalog_manager.cc:1347] Generated new cluster ID: 5364be1178154770b89e609421a9283c
I20260924 10:01:16.691807 23014 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:01:16.708626 23003 sys_catalog.cc:455] T 00000000000000000000000000000000 P 015aa72301464c2a8c177c26410e6efd [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "60762ee24c654fd99f78921bb0ddec54" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60762ee24c654fd99f78921bb0ddec54" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41177 } } peers { permanent_uuid: "bbf7177252cd496d8b9ad9b7ada894e8" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45793 } } peers { permanent_uuid: "015aa72301464c2a8c177c26410e6efd" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41439 } } }
I20260924 10:01:16.708909 23003 sys_catalog.cc:458] T 00000000000000000000000000000000 P 015aa72301464c2a8c177c26410e6efd [sys.catalog]: This master's current role is: FOLLOWER
I20260924 10:01:16.731995 23014 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:01:16.734488 23014 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:01:16.734704 22907 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41439 } attempt: 5
Sep 24 10:01:16 dist-test-slave-w8d2 krb5kdc[22753](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244075, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260924 10:01:16.752597 22909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:16.735258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41439 (local address 127.0.0.1:37844)
0924 10:01:16.735476 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0924 10:01:16.735495 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 10:01:16.735627 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:01:16.736053 (+ 426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:01:16.736058 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:01:16.736599 (+ 541us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:01:16.736795 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:01:16.736807 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:01:16.738419 (+ 1612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:01:16.738424 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:01:16.739830 (+ 1406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:01:16.739844 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:01:16.740769 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:01:16.740775 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:01:16.740980 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:01:16.741617 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:01:16.741641 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:01:16.744236 (+ 2595us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:01:16.748044 (+ 3808us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:01:16.748049 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:01:16.748052 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:01:16.748537 (+ 485us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:01:16.750694 (+ 2157us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:01:16.750699 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:01:16.750702 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:01:16.750873 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:01:16.751604 (+ 731us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:01:16.751610 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:01:16.751787 (+ 177us) client_negotiation.cc:769] Sending connection context
0924 10:01:16.751942 (+ 155us) client_negotiation.cc:240] Negotiation successful
0924 10:01:16.752224 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123}
I20260924 10:01:16.753520 22985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:16.735297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37844 (local address 127.0.97.60:41439)
0924 10:01:16.735453 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0924 10:01:16.735459 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:01:16.735548 (+ 89us) server_negotiation.cc:407] Connection header received
0924 10:01:16.735738 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:01:16.735742 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:01:16.735810 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:01:16.735890 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:01:16.736939 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:16.737970 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:16.739964 (+ 1994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:16.740634 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:16.740755 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:01:16.744408 (+ 3653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:01:16.744414 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:01:16.744419 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:01:16.744452 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:01:16.747931 (+ 3479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:01:16.750326 (+ 2395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:01:16.750332 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:01:16.750336 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:01:16.750494 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:01:16.751083 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:01:16.751088 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:01:16.751091 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:01:16.751483 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:01:16.751634 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0924 10:01:16.752970 (+ 1336us) server_negotiation.cc:299] Negotiation successful
0924 10:01:16.753206 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 10:01:16.768201 22907 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bbf7177252cd496d8b9ad9b7ada894e8: Bootstrap starting.
I20260924 10:01:16.769851 23014 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 60762ee24c654fd99f78921bb0ddec54: Generated new TSK 0
I20260924 10:01:16.770920 23014 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 10:01:16.774101 22907 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bbf7177252cd496d8b9ad9b7ada894e8: Neither blocks nor log segments found. Creating new log.
I20260924 10:01:16.775137 22907 log.cc:824] T 00000000000000000000000000000000 P bbf7177252cd496d8b9ad9b7ada894e8: Log is configured to *not* fsync() on all Append() calls
I20260924 10:01:16.761579 22914 master_runner.cc:186] Error getting master registration for 127.0.97.61:45793: OK, instance_id { permanent_uuid: "bbf7177252cd496d8b9ad9b7ada894e8" instance_seqno: 1790244076018892 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 10:01:16.783161 22914 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 10:01:16.786399 23028 catalog_manager.cc:1259] Loaded cluster ID: 5364be1178154770b89e609421a9283c
I20260924 10:01:16.787063 23028 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 015aa72301464c2a8c177c26410e6efd: loading cluster ID for follower catalog manager: success
I20260924 10:01:16.792747 23028 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 015aa72301464c2a8c177c26410e6efd: acquiring CA information for follower catalog manager: success
I20260924 10:01:16.801505 22907 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bbf7177252cd496d8b9ad9b7ada894e8: No bootstrap required, opened a new log
I20260924 10:01:16.795202 23028 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 015aa72301464c2a8c177c26410e6efd: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 10:01:16.821638 22907 raft_consensus.cc:348] T 00000000000000000000000000000000 P bbf7177252cd496d8b9ad9b7ada894e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60762ee24c654fd99f78921bb0ddec54" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41177 } } peers { permanent_uuid: "bbf7177252cd496d8b9ad9b7ada894e8" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45793 } } peers { permanent_uuid: "015aa72301464c2a8c177c26410e6efd" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41439 } }
I20260924 10:01:16.822108 22907 raft_consensus.cc:729] T 00000000000000000000000000000000 P bbf7177252cd496d8b9ad9b7ada894e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbf7177252cd496d8b9ad9b7ada894e8, State: Initialized, Role: FOLLOWER
I20260924 10:01:16.822671 22907 consensus_queue.cc:260] T 00000000000000000000000000000000 P bbf7177252cd496d8b9ad9b7ada894e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60762ee24c654fd99f78921bb0ddec54" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41177 } } peers { permanent_uuid: "bbf7177252cd496d8b9ad9b7ada894e8" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45793 } } peers { permanent_uuid: "015aa72301464c2a8c177c26410e6efd" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41439 } }
I20260924 10:01:16.824059 23032 sys_catalog.cc:455] T 00000000000000000000000000000000 P bbf7177252cd496d8b9ad9b7ada894e8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60762ee24c654fd99f78921bb0ddec54" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41177 } } peers { permanent_uuid: "bbf7177252cd496d8b9ad9b7ada894e8" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45793 } } peers { permanent_uuid: "015aa72301464c2a8c177c26410e6efd" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41439 } } }
I20260924 10:01:16.824811 23032 sys_catalog.cc:458] T 00000000000000000000000000000000 P bbf7177252cd496d8b9ad9b7ada894e8 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 10:01:16.832162 22907 sys_catalog.cc:564] T 00000000000000000000000000000000 P bbf7177252cd496d8b9ad9b7ada894e8 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:01:16.835413 22882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "60762ee24c654fd99f78921bb0ddec54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "bbf7177252cd496d8b9ad9b7ada894e8"
I20260924 10:01:16.835892 22882 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bbf7177252cd496d8b9ad9b7ada894e8 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:01:16.839864 22882 raft_consensus.cc:2454] T 00000000000000000000000000000000 P bbf7177252cd496d8b9ad9b7ada894e8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60762ee24c654fd99f78921bb0ddec54 in term 1.
I20260924 10:01:16.840615 22881 raft_consensus.cc:1261] T 00000000000000000000000000000000 P bbf7177252cd496d8b9ad9b7ada894e8 [term 1 FOLLOWER]: Refusing update from remote peer 60762ee24c654fd99f78921bb0ddec54: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:01:16.841653 23009 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 60762ee24c654fd99f78921bb0ddec54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf7177252cd496d8b9ad9b7ada894e8" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45793 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:01:16.883694 23032 sys_catalog.cc:455] T 00000000000000000000000000000000 P bbf7177252cd496d8b9ad9b7ada894e8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 60762ee24c654fd99f78921bb0ddec54. Latest consensus state: current_term: 1 leader_uuid: "60762ee24c654fd99f78921bb0ddec54" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60762ee24c654fd99f78921bb0ddec54" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41177 } } peers { permanent_uuid: "bbf7177252cd496d8b9ad9b7ada894e8" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45793 } } peers { permanent_uuid: "015aa72301464c2a8c177c26410e6efd" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41439 } } }
I20260924 10:01:16.884021 23032 sys_catalog.cc:458] T 00000000000000000000000000000000 P bbf7177252cd496d8b9ad9b7ada894e8 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 10:01:16.884460 23042 sys_catalog.cc:455] T 00000000000000000000000000000000 P bbf7177252cd496d8b9ad9b7ada894e8 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "60762ee24c654fd99f78921bb0ddec54" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60762ee24c654fd99f78921bb0ddec54" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41177 } } peers { permanent_uuid: "bbf7177252cd496d8b9ad9b7ada894e8" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45793 } } peers { permanent_uuid: "015aa72301464c2a8c177c26410e6efd" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41439 } } }
I20260924 10:01:16.885669 23042 sys_catalog.cc:458] T 00000000000000000000000000000000 P bbf7177252cd496d8b9ad9b7ada894e8 [sys.catalog]: This master's current role is: FOLLOWER
W20260924 10:01:16.897074 388 mini_kdc.cc:348] Time spent kdestroy: real 0.115s user 0.000s sys 0.021s
W20260924 10:01:16.932878 22836 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:01:16.930240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57020 (local address 127.0.97.62:41177)
0924 10:01:16.930466 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0924 10:01:16.930472 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:01:16.931007 (+ 535us) server_negotiation.cc:407] Connection header received
0924 10:01:16.931292 (+ 285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:01:16.931297 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:01:16.931383 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:01:16.931457 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:01:16.932624 (+ 1167us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57020: BlockingRecv error: recv got EOF from 127.0.0.1:57020 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260924 10:01:16.935158 22911 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:01:16.931432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46630 (local address 127.0.97.61:45793)
0924 10:01:16.931595 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0924 10:01:16.931601 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:01:16.933646 (+ 2045us) server_negotiation.cc:407] Connection header received
0924 10:01:16.933871 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:01:16.933877 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:01:16.933961 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:01:16.934025 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:01:16.934926 (+ 901us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46630: BlockingRecv error: recv got EOF from 127.0.0.1:46630 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 10:01:16.938555 23049 catalog_manager.cc:1259] Loaded cluster ID: 5364be1178154770b89e609421a9283c
I20260924 10:01:16.938661 23049 catalog_manager.cc:1552] T 00000000000000000000000000000000 P bbf7177252cd496d8b9ad9b7ada894e8: loading cluster ID for follower catalog manager: success
I20260924 10:01:16.942636 23049 catalog_manager.cc:1574] T 00000000000000000000000000000000 P bbf7177252cd496d8b9ad9b7ada894e8: acquiring CA information for follower catalog manager: success
I20260924 10:01:16.945122 23049 catalog_manager.cc:1602] T 00000000000000000000000000000000 P bbf7177252cd496d8b9ad9b7ada894e8: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260924 10:01:16.945798 22985 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:01:16.943168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37858 (local address 127.0.97.60:41439)
0924 10:01:16.943368 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0924 10:01:16.943374 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:01:16.944065 (+ 691us) server_negotiation.cc:407] Connection header received
0924 10:01:16.944320 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:01:16.944326 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:01:16.944407 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:01:16.944478 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:01:16.945551 (+ 1073us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37858: BlockingRecv error: recv got EOF from 127.0.0.1:37858 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
I20260924 10:01:17.007185 22911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:16.986227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46642 (local address 127.0.97.61:45793)
0924 10:01:16.986438 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0924 10:01:16.986445 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:01:16.999797 (+ 13352us) server_negotiation.cc:407] Connection header received
0924 10:01:17.000032 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:01:17.000037 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:01:17.000261 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:01:17.000419 (+ 158us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:01:17.001071 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:17.002236 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:17.004267 (+ 2031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:17.004914 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:17.005009 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:01:17.005706 (+ 697us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:01:17.006366 (+ 660us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:01:17.006477 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0924 10:01:17.006753 (+ 276us) server_negotiation.cc:299] Negotiation successful
0924 10:01:17.006842 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260924 10:01:17.062078 22985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:17.043785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37866 (local address 127.0.97.60:41439)
0924 10:01:17.043961 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0924 10:01:17.043967 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:01:17.054634 (+ 10667us) server_negotiation.cc:407] Connection header received
0924 10:01:17.055169 (+ 535us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:01:17.055175 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:01:17.055427 (+ 252us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:01:17.055571 (+ 144us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:01:17.056168 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:17.057280 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:17.059251 (+ 1971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:17.059877 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:17.059971 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:01:17.060675 (+ 704us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:01:17.061327 (+ 652us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:01:17.061431 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0924 10:01:17.061650 (+ 219us) server_negotiation.cc:299] Negotiation successful
0924 10:01:17.061738 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 10:01:17.120754 22836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:17.098305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57024 (local address 127.0.97.62:41177)
0924 10:01:17.098512 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0924 10:01:17.098519 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:01:17.109313 (+ 10794us) server_negotiation.cc:407] Connection header received
0924 10:01:17.110856 (+ 1543us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:01:17.110863 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:01:17.111096 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:01:17.111255 (+ 159us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:01:17.112433 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:17.113620 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:17.117560 (+ 3940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:17.118174 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:17.118268 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:01:17.119464 (+ 1196us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:01:17.119910 (+ 446us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:01:17.120001 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0924 10:01:17.120332 (+ 331us) server_negotiation.cc:299] Negotiation successful
0924 10:01:17.120427 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260924 10:01:17.130492 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu pid 22766
I20260924 10:01:17.131835 22828 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:01:17.132678 22985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:17.125483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37868 (local address 127.0.97.60:41439)
0924 10:01:17.125681 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0924 10:01:17.125687 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:01:17.126257 (+ 570us) server_negotiation.cc:407] Connection header received
0924 10:01:17.126326 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:01:17.126331 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:01:17.126407 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:01:17.126483 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:01:17.127120 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:17.128162 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:17.130015 (+ 1853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:17.130595 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:17.130712 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:01:17.131314 (+ 602us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:01:17.131457 (+ 143us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:01:17.131535 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0924 10:01:17.132315 (+ 780us) server_negotiation.cc:299] Negotiation successful
0924 10:01:17.132397 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260924 10:01:17.135456 22911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:17.109995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46658 (local address 127.0.97.61:45793)
0924 10:01:17.110174 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0924 10:01:17.110180 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:01:17.121488 (+ 11308us) server_negotiation.cc:407] Connection header received
0924 10:01:17.122280 (+ 792us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:01:17.122286 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:01:17.122398 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:01:17.122469 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:01:17.123359 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:17.124475 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:17.133038 (+ 8563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:17.133756 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:17.133896 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:01:17.134702 (+ 806us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:01:17.134854 (+ 152us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:01:17.134955 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0924 10:01:17.135066 (+ 111us) server_negotiation.cc:299] Negotiation successful
0924 10:01:17.135153 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 10:01:17.337440 23009 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60762ee24c654fd99f78921bb0ddec54 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "60762ee24c654fd99f78921bb0ddec54" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60762ee24c654fd99f78921bb0ddec54" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41177 } } peers { permanent_uuid: "bbf7177252cd496d8b9ad9b7ada894e8" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45793 } } peers { permanent_uuid: "015aa72301464c2a8c177c26410e6efd" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41439 } } }
I20260924 10:01:17.337754 23009 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60762ee24c654fd99f78921bb0ddec54 [sys.catalog]: This master's current role is: LEADER
I20260924 10:01:17.340229 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu with pid 22766
I20260924 10:01:17.367290 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu pid 22840
I20260924 10:01:17.368328 22902 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:01:17.473453 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu with pid 22840
I20260924 10:01:17.490089 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu pid 22914
I20260924 10:01:17.491088 22976 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:01:17.593600 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu with pid 22914
2026-09-24T10:01:17Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (3700 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790243735634835-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:01:17 dist-test-slave-w8d2 krb5kdc[23079](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:01:17 dist-test-slave-w8d2 krb5kdc[23079](info): set up 2 sockets
Sep 24 10:01:17 dist-test-slave-w8d2 krb5kdc[23079](info): commencing operation
krb5kdc: starting...
W20260924 10:01:18.607550 388 mini_kdc.cc:121] Time spent starting KDC: real 0.978s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790243735634835-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790243735634835-388-0/krb5kdc/test-user.keytab.
Sep 24 10:01:18 dist-test-slave-w8d2 krb5kdc[23079](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244078, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:01:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:01:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790243735634835-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790243735634835-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260924 10:01:18.849989 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
/tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790243735634835-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790243735634835-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790243735634835-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790243735634835-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38567
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790243735634835-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:38567,127.0.97.61:46191,127.0.97.60:44999
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790243735634835-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790243735634835-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790243735634835-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:01:19.164914 23092 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 10:01:19.165259 23092 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:01:19.165359 23092 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:01:19.165472 23092 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:01:19.172645 23092 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:01:19.172741 23092 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:01:19.172818 23092 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:01:19.172881 23092 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:01:19.172972 23092 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:01:19.183465 23092 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790243735634835-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790243735634835-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:38567,127.0.97.61:46191,127.0.97.60:44999
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790243735634835-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38567
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790243735634835-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790243735634835-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:01:19.184969 23092 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:01:19.186291 23092 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:01:19.197244 23098 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:01:19.197306 23100 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:01:19.197302 23097 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:01:19.198048 23092 server_base.cc:1034] running on GCE node
I20260924 10:01:19.198717 23092 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:01:19.200088 23092 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:01:19.201287 23092 hybrid_clock.cc:648] HybridClock initialized: now 1790244079201232 us; error 28 us; skew 500 ppm
I20260924 10:01:19.201579 23092 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:01:19 dist-test-slave-w8d2 krb5kdc[23079](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244079, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:01:19.208213 23092 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260924 10:01:19.209710 23092 webserver.cc:460] Webserver started at http://127.0.97.62:37329/ using document root <none> and password file <none>
I20260924 10:01:19.210438 23092 fs_manager.cc:359] Metadata directory not provided
I20260924 10:01:19.210546 23092 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:01:19.210847 23092 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:01:19.213867 23092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790243735634835-388-0/minicluster-data/master-0/data/instance:
uuid: "1bdfe206f9fe4614ac8319ee70514d24"
format_stamp: "Formatted at 2026-09-24 10:01:19 on dist-test-slave-w8d2"
server_key: "c2cf52f7795b3df926556e526c008934"
server_key_iv: "92a088d0006b9b8ffb53cab3010d89ee"
server_key_version: "encryptionkey@0"
I20260924 10:01:19.214653 23092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790243735634835-388-0/minicluster-data/master-0/wal/instance:
uuid: "1bdfe206f9fe4614ac8319ee70514d24"
format_stamp: "Formatted at 2026-09-24 10:01:19 on dist-test-slave-w8d2"
server_key: "c2cf52f7795b3df926556e526c008934"
server_key_iv: "92a088d0006b9b8ffb53cab3010d89ee"
server_key_version: "encryptionkey@0"
I20260924 10:01:19.219875 23092 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260924 10:01:19.223311 23107 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:01:19.224648 23092 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 10:01:19.224798 23092 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790243735634835-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790243735634835-388-0/minicluster-data/master-0/wal
uuid: "1bdfe206f9fe4614ac8319ee70514d24"
format_stamp: "Formatted at 2026-09-24 10:01:19 on dist-test-slave-w8d2"
server_key: "c2cf52f7795b3df926556e526c008934"
server_key_iv: "92a088d0006b9b8ffb53cab3010d89ee"
server_key_version: "encryptionkey@0"
I20260924 10:01:19.224992 23092 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790243735634835-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790243735634835-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790243735634835-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:01:19.236608 23092 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:01:19.240620 23092 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:01:19.240852 23092 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:01:19.257594 23092 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:38567
I20260924 10:01:19.257614 23158 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:38567 every 8 connection(s)
I20260924 10:01:19.258882 23092 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790243735634835-388-0/minicluster-data/master-0/data/info.pb
I20260924 10:01:19.260368 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu as pid 23092
I20260924 10:01:19.260535 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790243735634835-388-0/minicluster-data/master-0/wal/instance
I20260924 10:01:19.260931 388 external_mini_cluster.cc:1428] Setting key e8e578dd537117d30c7f4478462aa31e
I20260924 10:01:19.262532 23159 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:01:19.269475 23159 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38567 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.97.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.61@KRBTEST.COM" created.
Sep 24 10:01:19 dist-test-slave-w8d2 krb5kdc[23079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244079, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 10:01:19.293660 23162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:19.273024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38567 (local address 127.0.0.1:56942)
0924 10:01:19.273794 (+ 770us) negotiation.cc:107] Waiting for socket to connect
0924 10:01:19.273845 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0924 10:01:19.275860 (+ 2015us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:01:19.276752 (+ 892us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:01:19.276756 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:01:19.277279 (+ 523us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:01:19.277603 (+ 324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:01:19.277619 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:01:19.279665 (+ 2046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:01:19.279669 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:01:19.280686 (+ 1017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:01:19.280697 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:01:19.281876 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:01:19.281881 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:01:19.282030 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:01:19.282591 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:01:19.282612 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:01:19.286876 (+ 4264us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:01:19.289840 (+ 2964us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:01:19.289851 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:01:19.289864 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:01:19.290342 (+ 478us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:01:19.290730 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:01:19.290735 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:01:19.290738 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:01:19.290942 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:01:19.291698 (+ 756us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:01:19.291707 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:01:19.291939 (+ 232us) client_negotiation.cc:769] Sending connection context
0924 10:01:19.292199 (+ 260us) client_negotiation.cc:240] Negotiation successful
0924 10:01:19.292748 (+ 549us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":510,"mutex_wait_us":81,"thread_start_us":313,"threads_started":1}
I20260924 10:01:19.293713 23163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:19.274215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56942 (local address 127.0.97.62:38567)
0924 10:01:19.274872 (+ 657us) server_negotiation.cc:206] Beginning negotiation
0924 10:01:19.274880 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 10:01:19.274928 (+ 48us) server_negotiation.cc:407] Connection header received
0924 10:01:19.276215 (+ 1287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:01:19.276221 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:01:19.276542 (+ 321us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:01:19.276749 (+ 207us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:01:19.277798 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:19.279531 (+ 1733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:19.280823 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:19.281670 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:19.281847 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:01:19.287094 (+ 5247us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:01:19.287102 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:01:19.287115 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:01:19.287148 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:01:19.289708 (+ 2560us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:01:19.290459 (+ 751us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:01:19.290465 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:01:19.290467 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:01:19.290627 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:01:19.291033 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:01:19.291036 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:01:19.291038 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:01:19.291574 (+ 536us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:01:19.291723 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0924 10:01:19.292299 (+ 576us) server_negotiation.cc:299] Negotiation successful
0924 10:01:19.292575 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":447,"thread_start_us":312,"threads_started":1}
I20260924 10:01:19.296828 23159 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46191 } has no permanent_uuid. Determining permanent_uuid...
I20260924 10:01:19.297904 23162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:19.297499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:46191 (local address 127.0.0.1:35190)
0924 10:01:19.297664 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0924 10:01:19.297778 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:46191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260924 10:01:19.298233 23111 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.61:46191: connect: Connection refused (error 111)
I20260924 10:01:19.299814 23162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:19.299431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:46191 (local address 127.0.0.1:35202)
0924 10:01:19.299582 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0924 10:01:19.299670 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:46191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260924 10:01:19.300028 23159 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:46191: Network error: Client connection negotiation failed: client connection to 127.0.97.61:46191: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790243735634835-388-0/krb5kdc/kudu_127.0.97.61.keytab.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790243735634835-388-0/krb5kdc/kudu_127.0.97.61.keytab.
I20260924 10:01:19.314381 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
/tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790243735634835-388-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790243735634835-388-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790243735634835-388-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790243735634835-388-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:46191
--webserver_interface=127.0.97.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790243735634835-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:38567,127.0.97.61:46191,127.0.97.60:44999
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790243735634835-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790243735634835-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790243735634835-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 10:01:19.349385 23159 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46191 } attempt: 1
I20260924 10:01:19.350613 23162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:19.350106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:46191 (local address 127.0.0.1:35208)
0924 10:01:19.350325 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0924 10:01:19.350454 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:46191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260924 10:01:19.351665 23162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:19.351289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:46191 (local address 127.0.0.1:35216)
0924 10:01:19.351434 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0924 10:01:19.351517 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:46191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260924 10:01:19.351899 23159 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:46191: Network error: Client connection negotiation failed: client connection to 127.0.97.61:46191: connect: Connection refused (error 111)
I20260924 10:01:19.420163 23159 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46191 } attempt: 2
I20260924 10:01:19.421402 23162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:19.420849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:46191 (local address 127.0.0.1:35228)
0924 10:01:19.421114 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0924 10:01:19.421229 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:46191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260924 10:01:19.422487 23162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:19.422061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:46191 (local address 127.0.0.1:35236)
0924 10:01:19.422229 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0924 10:01:19.422334 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:46191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260924 10:01:19.422672 23159 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:46191: Network error: Client connection negotiation failed: client connection to 127.0.97.61:46191: connect: Connection refused (error 111)
I20260924 10:01:19.513954 23159 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46191 } attempt: 3
I20260924 10:01:19.515358 23162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:19.514720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:46191 (local address 127.0.0.1:35242)
0924 10:01:19.514973 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0924 10:01:19.515102 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:46191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260924 10:01:19.516414 23162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:19.516022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:46191 (local address 127.0.0.1:35248)
0924 10:01:19.516162 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0924 10:01:19.516259 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:46191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260924 10:01:19.516603 23159 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:46191: Network error: Client connection negotiation failed: client connection to 127.0.97.61:46191: connect: Connection refused (error 111)
W20260924 10:01:19.629297 23166 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 10:01:19.629632 23166 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:01:19.629735 23166 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:01:19.629849 23166 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:01:19.636977 23166 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:01:19.637072 23166 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:01:19.637158 23166 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:01:19.637220 23166 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:01:19.637305 23166 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:01:19.646996 23166 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790243735634835-388-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790243735634835-388-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:38567,127.0.97.61:46191,127.0.97.60:44999
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790243735634835-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:46191
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790243735634835-388-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.97.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790243735634835-388-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:01:19.648303 23166 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:01:19.649541 23166 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 10:01:19.659811 23159 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46191 } attempt: 4
W20260924 10:01:19.660588 23174 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:01:19.660645 23171 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:01:19.660676 23172 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:01:19.661113 23162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:19.660455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:46191 (local address 127.0.0.1:35252)
0924 10:01:19.660865 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0924 10:01:19.660960 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:46191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":267}
I20260924 10:01:19.662166 23162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:19.661758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:46191 (local address 127.0.0.1:35258)
0924 10:01:19.661899 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0924 10:01:19.661999 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:46191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260924 10:01:19.662228 23166 server_base.cc:1034] running on GCE node
W20260924 10:01:19.662355 23159 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:46191: Network error: Client connection negotiation failed: client connection to 127.0.97.61:46191: connect: Connection refused (error 111)
I20260924 10:01:19.663019 23166 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:01:19.664353 23166 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:01:19.665557 23166 hybrid_clock.cc:648] HybridClock initialized: now 1790244079665501 us; error 31 us; skew 500 ppm
I20260924 10:01:19.665865 23166 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:01:19 dist-test-slave-w8d2 krb5kdc[23079](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244079, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:01:19.668805 23166 init.cc:377] Logged in from keytab as kudu/127.0.97.61@KRBTEST.COM (short username kudu)
I20260924 10:01:19.670094 23166 webserver.cc:460] Webserver started at http://127.0.97.61:45735/ using document root <none> and password file <none>
I20260924 10:01:19.670742 23166 fs_manager.cc:359] Metadata directory not provided
I20260924 10:01:19.670825 23166 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:01:19.671066 23166 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:01:19.673900 23166 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790243735634835-388-0/minicluster-data/master-1/data/instance:
uuid: "31c2308dcf0147c8b752cb0f466f95f5"
format_stamp: "Formatted at 2026-09-24 10:01:19 on dist-test-slave-w8d2"
server_key: "d40f37bbdd735546bdebc600ab0bde8b"
server_key_iv: "ed863d4ca0cb619379796706a3337733"
server_key_version: "encryptionkey@0"
I20260924 10:01:19.674624 23166 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790243735634835-388-0/minicluster-data/master-1/wal/instance:
uuid: "31c2308dcf0147c8b752cb0f466f95f5"
format_stamp: "Formatted at 2026-09-24 10:01:19 on dist-test-slave-w8d2"
server_key: "d40f37bbdd735546bdebc600ab0bde8b"
server_key_iv: "ed863d4ca0cb619379796706a3337733"
server_key_version: "encryptionkey@0"
I20260924 10:01:19.679628 23166 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260924 10:01:19.682862 23181 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:01:19.684140 23166 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 10:01:19.684279 23166 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790243735634835-388-0/minicluster-data/master-1/data,/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790243735634835-388-0/minicluster-data/master-1/wal
uuid: "31c2308dcf0147c8b752cb0f466f95f5"
format_stamp: "Formatted at 2026-09-24 10:01:19 on dist-test-slave-w8d2"
server_key: "d40f37bbdd735546bdebc600ab0bde8b"
server_key_iv: "ed863d4ca0cb619379796706a3337733"
server_key_version: "encryptionkey@0"
I20260924 10:01:19.684470 23166 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790243735634835-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790243735634835-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790243735634835-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:01:19.706782 23166 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:01:19.711057 23166 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:01:19.711297 23166 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:01:19.727550 23166 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:46191
I20260924 10:01:19.727568 23232 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:46191 every 8 connection(s)
I20260924 10:01:19.728950 23166 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790243735634835-388-0/minicluster-data/master-1/data/info.pb
I20260924 10:01:19.732538 23233 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:01:19.737735 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu as pid 23166
I20260924 10:01:19.737885 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790243735634835-388-0/minicluster-data/master-1/wal/instance
I20260924 10:01:19.738306 388 external_mini_cluster.cc:1428] Setting key fe251d91f7597f6c97c1ec2a8121f4a1
I20260924 10:01:19.738796 23233 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38567 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 10:01:19 dist-test-slave-w8d2 krb5kdc[23079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244079, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 10:01:19.759100 23163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:19.741243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56956 (local address 127.0.97.62:38567)
0924 10:01:19.741432 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0924 10:01:19.741437 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:01:19.742620 (+ 1183us) server_negotiation.cc:407] Connection header received
0924 10:01:19.744947 (+ 2327us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:01:19.744952 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:01:19.745022 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:01:19.745134 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:01:19.746953 (+ 1819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:19.747975 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:19.749426 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:19.749967 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:19.750080 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:01:19.753361 (+ 3281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:01:19.753367 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:01:19.753371 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:01:19.753410 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:01:19.756338 (+ 2928us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:01:19.756975 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:01:19.756980 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:01:19.756984 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:01:19.757117 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:01:19.757452 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:01:19.757456 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:01:19.757459 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:01:19.757794 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:01:19.757927 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0924 10:01:19.758544 (+ 617us) server_negotiation.cc:299] Negotiation successful
0924 10:01:19.758771 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 10:01:19.759915 23235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:19.741703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38567 (local address 127.0.0.1:56956)
0924 10:01:19.742435 (+ 732us) negotiation.cc:107] Waiting for socket to connect
0924 10:01:19.742489 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0924 10:01:19.744631 (+ 2142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:01:19.745380 (+ 749us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:01:19.745391 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:01:19.746058 (+ 667us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:01:19.746755 (+ 697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:01:19.746793 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:01:19.748099 (+ 1306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:01:19.748103 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:01:19.749285 (+ 1182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:01:19.749296 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:01:19.750137 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:01:19.750142 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:01:19.750330 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:01:19.750929 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:01:19.750950 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:01:19.753148 (+ 2198us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:01:19.756454 (+ 3306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:01:19.756461 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:01:19.756473 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:01:19.756879 (+ 406us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:01:19.757211 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:01:19.757214 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:01:19.757216 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:01:19.757365 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:01:19.757889 (+ 524us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:01:19.757894 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:01:19.758190 (+ 296us) client_negotiation.cc:769] Sending connection context
0924 10:01:19.758442 (+ 252us) client_negotiation.cc:240] Negotiation successful
0924 10:01:19.758839 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":285,"threads_started":1}
I20260924 10:01:19.761535 23233 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46191 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.97.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.60@KRBTEST.COM" created.
Sep 24 10:01:19 dist-test-slave-w8d2 krb5kdc[23079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244079, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260924 10:01:19.775342 23235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:19.762304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:46191 (local address 127.0.0.1:35262)
0924 10:01:19.762452 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0924 10:01:19.762470 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 10:01:19.762562 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:01:19.763560 (+ 998us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:01:19.763564 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:01:19.763893 (+ 329us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:01:19.764039 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:01:19.764048 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:01:19.765433 (+ 1385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:01:19.765439 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:01:19.766630 (+ 1191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:01:19.766642 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:01:19.767411 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:01:19.767420 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:01:19.767557 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:01:19.768049 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:01:19.768068 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:01:19.770086 (+ 2018us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:01:19.773030 (+ 2944us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:01:19.773035 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:01:19.773040 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:01:19.773388 (+ 348us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:01:19.773718 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:01:19.773722 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:01:19.773726 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:01:19.773857 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:01:19.774406 (+ 549us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:01:19.774411 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:01:19.774549 (+ 138us) client_negotiation.cc:769] Sending connection context
0924 10:01:19.774667 (+ 118us) client_negotiation.cc:240] Negotiation successful
0924 10:01:19.774935 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260924 10:01:19.775447 23237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:19.762549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35262 (local address 127.0.97.61:46191)
0924 10:01:19.763119 (+ 570us) server_negotiation.cc:206] Beginning negotiation
0924 10:01:19.763125 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:01:19.763149 (+ 24us) server_negotiation.cc:407] Connection header received
0924 10:01:19.763191 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:01:19.763197 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:01:19.763431 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:01:19.763548 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:01:19.764182 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:19.765286 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:19.766745 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:19.767270 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:19.767363 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:01:19.770274 (+ 2911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:01:19.770284 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:01:19.770297 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:01:19.770324 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:01:19.772916 (+ 2592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:01:19.773500 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:01:19.773506 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:01:19.773508 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:01:19.773612 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:01:19.773947 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:01:19.773952 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:01:19.773955 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:01:19.774283 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:01:19.774464 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0924 10:01:19.774803 (+ 339us) server_negotiation.cc:299] Negotiation successful
0924 10:01:19.775058 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":25,"server-negotiator.queue_time_us":436,"thread_start_us":313,"threads_started":1}
I20260924 10:01:19.777652 23233 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44999 } has no permanent_uuid. Determining permanent_uuid...
I20260924 10:01:19.778779 23235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:19.778333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44999 (local address 127.0.0.1:41070)
0924 10:01:19.778499 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0924 10:01:19.778612 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260924 10:01:19.779027 23184 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44999: connect: Connection refused (error 111)
I20260924 10:01:19.780486 23235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:19.780191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44999 (local address 127.0.0.1:41086)
0924 10:01:19.780320 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0924 10:01:19.780392 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260924 10:01:19.780673 23233 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:44999: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44999: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790243735634835-388-0/krb5kdc/kudu_127.0.97.60.keytab.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790243735634835-388-0/krb5kdc/kudu_127.0.97.60.keytab.
I20260924 10:01:19.792464 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
/tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790243735634835-388-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790243735634835-388-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790243735634835-388-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790243735634835-388-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:44999
--webserver_interface=127.0.97.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790243735634835-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:38567,127.0.97.61:46191,127.0.97.60:44999
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790243735634835-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790243735634835-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790243735634835-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 10:01:19.829928 23233 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44999 } attempt: 1
I20260924 10:01:19.830928 23235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:19.830512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44999 (local address 127.0.0.1:41098)
0924 10:01:19.830697 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0924 10:01:19.830781 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260924 10:01:19.831908 23235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:19.831604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44999 (local address 127.0.0.1:41110)
0924 10:01:19.831737 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0924 10:01:19.831808 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260924 10:01:19.832103 23233 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:44999: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44999: connect: Connection refused (error 111)
I20260924 10:01:19.900306 23233 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44999 } attempt: 2
I20260924 10:01:19.901458 23235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:19.901008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44999 (local address 127.0.0.1:41114)
0924 10:01:19.901214 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0924 10:01:19.901305 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260924 10:01:19.902384 23235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:19.902049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44999 (local address 127.0.0.1:41120)
0924 10:01:19.902204 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0924 10:01:19.902273 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260924 10:01:19.902575 23233 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:44999: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44999: connect: Connection refused (error 111)
I20260924 10:01:19.961647 23159 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46191 } attempt: 5
Sep 24 10:01:19 dist-test-slave-w8d2 krb5kdc[23079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244079, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260924 10:01:19.974522 23237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:19.962471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35268 (local address 127.0.97.61:46191)
0924 10:01:19.962659 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0924 10:01:19.962665 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:01:19.962686 (+ 21us) server_negotiation.cc:407] Connection header received
0924 10:01:19.962847 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:01:19.962852 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:01:19.962914 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:01:19.962998 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:01:19.963861 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:19.964543 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:19.965674 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:19.966145 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:19.966251 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:01:19.969515 (+ 3264us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:01:19.969520 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:01:19.969523 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:01:19.969556 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:01:19.972035 (+ 2479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:01:19.972828 (+ 793us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:01:19.972831 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:01:19.972834 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:01:19.972940 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:01:19.973311 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:01:19.973314 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:01:19.973316 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:01:19.973573 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:01:19.973675 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0924 10:01:19.974120 (+ 445us) server_negotiation.cc:299] Negotiation successful
0924 10:01:19.974281 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260924 10:01:19.974711 23162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:19.962365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:46191 (local address 127.0.0.1:35268)
0924 10:01:19.962553 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0924 10:01:19.962583 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0924 10:01:19.962713 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:01:19.963075 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:01:19.963080 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:01:19.963515 (+ 435us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:01:19.963706 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:01:19.963716 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:01:19.964684 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:01:19.964690 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:01:19.965518 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:01:19.965536 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:01:19.966284 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:01:19.966290 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:01:19.966476 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:01:19.967100 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:01:19.967117 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:01:19.969333 (+ 2216us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:01:19.972177 (+ 2844us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:01:19.972183 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:01:19.972187 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:01:19.972726 (+ 539us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:01:19.973039 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:01:19.973044 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:01:19.973048 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:01:19.973219 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:01:19.973685 (+ 466us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:01:19.973690 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:01:19.973874 (+ 184us) client_negotiation.cc:769] Sending connection context
0924 10:01:19.974045 (+ 171us) client_negotiation.cc:240] Negotiation successful
0924 10:01:19.974346 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61}
I20260924 10:01:19.975749 23159 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44999 } has no permanent_uuid. Determining permanent_uuid...
I20260924 10:01:19.976876 23162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:19.976508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44999 (local address 127.0.0.1:41128)
0924 10:01:19.976667 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0924 10:01:19.976740 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260924 10:01:19.977780 23162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:19.977470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44999 (local address 127.0.0.1:41132)
0924 10:01:19.977607 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0924 10:01:19.977687 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260924 10:01:19.977967 23159 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:44999: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44999: connect: Connection refused (error 111)
I20260924 10:01:19.993762 23233 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44999 } attempt: 3
I20260924 10:01:19.994717 23235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:19.994286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44999 (local address 127.0.0.1:41136)
0924 10:01:19.994480 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0924 10:01:19.994569 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260924 10:01:19.995741 23235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:19.995290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44999 (local address 127.0.0.1:41148)
0924 10:01:19.995521 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0924 10:01:19.995630 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124}
W20260924 10:01:19.995929 23233 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:44999: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44999: connect: Connection refused (error 111)
I20260924 10:01:20.029170 23159 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44999 } attempt: 1
I20260924 10:01:20.030198 23162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:20.029756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44999 (local address 127.0.0.1:41160)
0924 10:01:20.029941 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0924 10:01:20.030037 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260924 10:01:20.031141 23162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:20.030828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44999 (local address 127.0.0.1:41168)
0924 10:01:20.030968 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0924 10:01:20.031039 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260924 10:01:20.031351 23159 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:44999: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44999: connect: Connection refused (error 111)
I20260924 10:01:20.099529 23159 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44999 } attempt: 2
I20260924 10:01:20.100600 23162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:20.100096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44999 (local address 127.0.0.1:41184)
0924 10:01:20.100307 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0924 10:01:20.100391 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260924 10:01:20.101541 23162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:20.101227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44999 (local address 127.0.0.1:41186)
0924 10:01:20.101370 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0924 10:01:20.101437 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260924 10:01:20.101732 23159 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:44999: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44999: connect: Connection refused (error 111)
W20260924 10:01:20.104573 23240 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 10:01:20.104892 23240 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:01:20.104984 23240 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:01:20.105105 23240 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:01:20.112099 23240 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:01:20.112177 23240 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:01:20.112264 23240 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:01:20.112330 23240 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:01:20.112409 23240 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:01:20.122080 23240 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790243735634835-388-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790243735634835-388-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:38567,127.0.97.61:46191,127.0.97.60:44999
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790243735634835-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:44999
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790243735634835-388-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.97.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790243735634835-388-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:01:20.123605 23240 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:01:20.124979 23240 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:01:20.136260 23245 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:01:20.136297 23246 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:01:20.136320 23248 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:01:20.136992 23240 server_base.cc:1034] running on GCE node
I20260924 10:01:20.137838 23240 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:01:20.139127 23233 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44999 } attempt: 4
I20260924 10:01:20.139338 23240 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:01:20.140269 23235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:20.139759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44999 (local address 127.0.0.1:41190)
0924 10:01:20.139988 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0924 10:01:20.140089 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260924 10:01:20.140554 23240 hybrid_clock.cc:648] HybridClock initialized: now 1790244080140482 us; error 44 us; skew 500 ppm
I20260924 10:01:20.140869 23240 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 10:01:20.141127 23235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:20.140814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44999 (local address 127.0.0.1:41192)
0924 10:01:20.140945 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0924 10:01:20.141025 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260924 10:01:20.141387 23233 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:44999: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44999: connect: Connection refused (error 111)
Sep 24 10:01:20 dist-test-slave-w8d2 krb5kdc[23079](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244080, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:01:20.143919 23240 init.cc:377] Logged in from keytab as kudu/127.0.97.60@KRBTEST.COM (short username kudu)
I20260924 10:01:20.145248 23240 webserver.cc:460] Webserver started at http://127.0.97.60:35817/ using document root <none> and password file <none>
I20260924 10:01:20.145922 23240 fs_manager.cc:359] Metadata directory not provided
I20260924 10:01:20.146006 23240 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:01:20.146257 23240 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:01:20.149119 23240 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790243735634835-388-0/minicluster-data/master-2/data/instance:
uuid: "7a9b1b5ca75d43308184b1105a3fee03"
format_stamp: "Formatted at 2026-09-24 10:01:20 on dist-test-slave-w8d2"
server_key: "9f6686aabada3aff1583db08a712b858"
server_key_iv: "43f93f23f830de7d531cc3f60ec03a13"
server_key_version: "encryptionkey@0"
I20260924 10:01:20.149827 23240 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790243735634835-388-0/minicluster-data/master-2/wal/instance:
uuid: "7a9b1b5ca75d43308184b1105a3fee03"
format_stamp: "Formatted at 2026-09-24 10:01:20 on dist-test-slave-w8d2"
server_key: "9f6686aabada3aff1583db08a712b858"
server_key_iv: "43f93f23f830de7d531cc3f60ec03a13"
server_key_version: "encryptionkey@0"
I20260924 10:01:20.154762 23240 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 10:01:20.158047 23255 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:01:20.159390 23240 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 10:01:20.159533 23240 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790243735634835-388-0/minicluster-data/master-2/data,/tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790243735634835-388-0/minicluster-data/master-2/wal
uuid: "7a9b1b5ca75d43308184b1105a3fee03"
format_stamp: "Formatted at 2026-09-24 10:01:20 on dist-test-slave-w8d2"
server_key: "9f6686aabada3aff1583db08a712b858"
server_key_iv: "43f93f23f830de7d531cc3f60ec03a13"
server_key_version: "encryptionkey@0"
I20260924 10:01:20.159706 23240 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790243735634835-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790243735634835-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790243735634835-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:01:20.197201 23240 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:01:20.201200 23240 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:01:20.201423 23240 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:01:20.207914 23159 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44999 } attempt: 3
I20260924 10:01:20.209043 23162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:20.208526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44999 (local address 127.0.0.1:41194)
0924 10:01:20.208739 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0924 10:01:20.208837 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260924 10:01:20.210330 23162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:20.209818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44999 (local address 127.0.0.1:41206)
0924 10:01:20.210096 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0924 10:01:20.210189 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182}
W20260924 10:01:20.210565 23159 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:44999: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44999: connect: Connection refused (error 111)
I20260924 10:01:20.218811 23240 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:44999
I20260924 10:01:20.218844 23306 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:44999 every 8 connection(s)
I20260924 10:01:20.220145 23240 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790243735634835-388-0/minicluster-data/master-2/data/info.pb
I20260924 10:01:20.223793 23307 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:01:20.225037 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu as pid 23240
I20260924 10:01:20.225180 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_ZmK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790243735634835-388-0/minicluster-data/master-2/wal/instance
I20260924 10:01:20.225598 388 external_mini_cluster.cc:1428] Setting key b54cac8090f010d53fa9f1228d389272
I20260924 10:01:20.232936 23307 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38567 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 10:01:20 dist-test-slave-w8d2 krb5kdc[23079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244078, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Sep 24 10:01:20 dist-test-slave-w8d2 krb5kdc[23079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244080, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 10:01:20.247625 23310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:20.227379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41218 (local address 127.0.97.60:44999)
0924 10:01:20.228163 (+ 784us) server_negotiation.cc:206] Beginning negotiation
0924 10:01:20.228179 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 10:01:20.228228 (+ 49us) server_negotiation.cc:407] Connection header received
0924 10:01:20.229880 (+ 1652us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:01:20.229908 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:01:20.230452 (+ 544us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:01:20.230900 (+ 448us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:01:20.232206 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:20.234117 (+ 1911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:20.235487 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:20.236312 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:20.236478 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:01:20.240134 (+ 3656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:01:20.240141 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:01:20.240151 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:01:20.240184 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:01:20.243573 (+ 3389us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:01:20.244321 (+ 748us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:01:20.244327 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:01:20.244333 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:01:20.244475 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:01:20.244870 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:01:20.244893 (+ 23us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:01:20.244896 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:01:20.245432 (+ 536us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:01:20.245650 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0924 10:01:20.246108 (+ 458us) server_negotiation.cc:299] Negotiation successful
0924 10:01:20.246427 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":515,"thread_start_us":300,"threads_started":1}
I20260924 10:01:20.249639 23311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:20.235779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38567 (local address 127.0.0.1:56970)
0924 10:01:20.236575 (+ 796us) negotiation.cc:107] Waiting for socket to connect
0924 10:01:20.236615 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 10:01:20.236765 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:01:20.237136 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:01:20.237143 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:01:20.237710 (+ 567us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:01:20.237936 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:01:20.237947 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:01:20.238853 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:01:20.238858 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:01:20.239907 (+ 1049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:01:20.239918 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:01:20.240697 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:01:20.240704 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:01:20.240840 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:01:20.241351 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:01:20.241377 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:01:20.243931 (+ 2554us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:01:20.247136 (+ 3205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:01:20.247146 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:01:20.247153 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:01:20.247718 (+ 565us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:01:20.248152 (+ 434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:01:20.248158 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:01:20.248161 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:01:20.248323 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:01:20.248818 (+ 495us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:01:20.248825 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:01:20.249008 (+ 183us) client_negotiation.cc:769] Sending connection context
0924 10:01:20.249150 (+ 142us) client_negotiation.cc:240] Negotiation successful
0924 10:01:20.249354 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":568,"thread_start_us":415,"threads_started":1}
I20260924 10:01:20.250147 23163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:20.235795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56970 (local address 127.0.97.62:38567)
0924 10:01:20.235997 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0924 10:01:20.236004 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:01:20.236700 (+ 696us) server_negotiation.cc:407] Connection header received
0924 10:01:20.236886 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:01:20.236891 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:01:20.236974 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:01:20.237059 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:01:20.238078 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:20.238718 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:20.240049 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:20.240511 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:20.240625 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:01:20.244129 (+ 3504us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:01:20.244135 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:01:20.244140 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:01:20.244193 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:01:20.247019 (+ 2826us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:01:20.247879 (+ 860us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:01:20.247884 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:01:20.247888 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:01:20.248033 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:01:20.248436 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:01:20.248440 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:01:20.248442 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:01:20.248686 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:01:20.248843 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0924 10:01:20.249300 (+ 457us) server_negotiation.cc:299] Negotiation successful
0924 10:01:20.249526 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 10:01:20.251529 23307 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46191 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 10:01:20 dist-test-slave-w8d2 krb5kdc[23079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244080, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
Sep 24 10:01:20 dist-test-slave-w8d2 krb5kdc[23079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244078, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 10:01:20.264514 23237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:20.252195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35278 (local address 127.0.97.61:46191)
0924 10:01:20.252371 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0924 10:01:20.252375 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:01:20.252392 (+ 17us) server_negotiation.cc:407] Connection header received
0924 10:01:20.252437 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:01:20.252441 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:01:20.252496 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:01:20.252574 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:01:20.253199 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:20.253915 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:20.255013 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:20.255458 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:20.255560 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:01:20.258640 (+ 3080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:01:20.258645 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:01:20.258648 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:01:20.258678 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:01:20.261651 (+ 2973us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:01:20.262210 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:01:20.262216 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:01:20.262220 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:01:20.262397 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:01:20.262826 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:01:20.262832 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:01:20.262836 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:01:20.263156 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:01:20.263308 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0924 10:01:20.263938 (+ 630us) server_negotiation.cc:299] Negotiation successful
0924 10:01:20.264154 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 10:01:20.265293 23311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:20.252130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:46191 (local address 127.0.0.1:35278)
0924 10:01:20.252278 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0924 10:01:20.252294 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 10:01:20.252366 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:01:20.252595 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:01:20.252597 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:01:20.252906 (+ 309us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:01:20.253054 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:01:20.253064 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:01:20.254052 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:01:20.254056 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:01:20.254878 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:01:20.254888 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:01:20.255593 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:01:20.255598 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:01:20.255767 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:01:20.256236 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:01:20.256252 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:01:20.258475 (+ 2223us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:01:20.261763 (+ 3288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:01:20.261768 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:01:20.261773 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:01:20.262103 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:01:20.262513 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:01:20.262518 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:01:20.262521 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:01:20.262695 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:01:20.263406 (+ 711us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:01:20.263411 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:01:20.263639 (+ 228us) client_negotiation.cc:769] Sending connection context
0924 10:01:20.264694 (+ 1055us) client_negotiation.cc:240] Negotiation successful
0924 10:01:20.264977 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260924 10:01:20.266319 23307 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44999 } has no permanent_uuid. Determining permanent_uuid...
I20260924 10:01:20.266467 23163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:20.250841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56972 (local address 127.0.97.62:38567)
0924 10:01:20.251006 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0924 10:01:20.251011 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:01:20.251047 (+ 36us) server_negotiation.cc:407] Connection header received
0924 10:01:20.251206 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:01:20.251211 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:01:20.251303 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:01:20.251394 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:01:20.253039 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:20.253781 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:20.254993 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:20.255471 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:20.255581 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:01:20.259976 (+ 4395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:01:20.259983 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:01:20.259987 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:01:20.260023 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:01:20.263687 (+ 3664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:01:20.264364 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:01:20.264373 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:01:20.264377 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:01:20.264511 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:01:20.264879 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:01:20.264884 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:01:20.264887 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:01:20.265186 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:01:20.265671 (+ 485us) server_negotiation.cc:1035] Waiting for connection context
0924 10:01:20.265926 (+ 255us) server_negotiation.cc:299] Negotiation successful
0924 10:01:20.266134 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
Sep 24 10:01:20 dist-test-slave-w8d2 krb5kdc[23079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244080, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Sep 24 10:01:20 dist-test-slave-w8d2 krb5kdc[23079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244078, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260924 10:01:20.280334 23310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:20.267316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41224 (local address 127.0.97.60:44999)
0924 10:01:20.267446 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0924 10:01:20.267451 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:01:20.267467 (+ 16us) server_negotiation.cc:407] Connection header received
0924 10:01:20.267596 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:01:20.267601 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:01:20.267650 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:01:20.267720 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:01:20.268449 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:20.269464 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:20.271067 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:20.271732 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:20.271867 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:01:20.274920 (+ 3053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:01:20.274927 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:01:20.274932 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:01:20.274971 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:01:20.277828 (+ 2857us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:01:20.278510 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:01:20.278514 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:01:20.278516 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:01:20.278626 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:01:20.278994 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:01:20.278999 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:01:20.279001 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:01:20.279372 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:01:20.279480 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0924 10:01:20.279882 (+ 402us) server_negotiation.cc:299] Negotiation successful
0924 10:01:20.280057 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 10:01:20.280658 23311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:20.267184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44999 (local address 127.0.0.1:41224)
0924 10:01:20.267375 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0924 10:01:20.267392 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 10:01:20.267497 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:01:20.267747 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:01:20.267752 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:01:20.268148 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:01:20.268309 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:01:20.268320 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:01:20.269607 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:01:20.269611 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:01:20.270919 (+ 1308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:01:20.270933 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:01:20.271902 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:01:20.271906 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:01:20.272075 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:01:20.272589 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:01:20.272606 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:01:20.274729 (+ 2123us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:01:20.277957 (+ 3228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:01:20.277962 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:01:20.277966 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:01:20.278396 (+ 430us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:01:20.278730 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:01:20.278735 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:01:20.278739 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:01:20.278889 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:01:20.279482 (+ 593us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:01:20.279487 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:01:20.279673 (+ 186us) client_negotiation.cc:769] Sending connection context
0924 10:01:20.279825 (+ 152us) client_negotiation.cc:240] Negotiation successful
0924 10:01:20.280261 (+ 436us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":87,"mutex_wait_us":93}
I20260924 10:01:20.283224 23237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:20.268022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35284 (local address 127.0.97.61:46191)
0924 10:01:20.268177 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0924 10:01:20.268182 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:01:20.268198 (+ 16us) server_negotiation.cc:407] Connection header received
0924 10:01:20.268243 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:01:20.268248 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:01:20.268308 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:01:20.268403 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:01:20.269760 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:20.270818 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:20.272400 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:20.272800 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:20.272908 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:01:20.277580 (+ 4672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:01:20.277585 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:01:20.277588 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:01:20.277623 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:01:20.280724 (+ 3101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:01:20.281430 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:01:20.281435 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:01:20.281438 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:01:20.281601 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:01:20.282019 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:01:20.282023 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:01:20.282025 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:01:20.282279 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:01:20.282484 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0924 10:01:20.282826 (+ 342us) server_negotiation.cc:299] Negotiation successful
0924 10:01:20.282994 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 10:01:20.286108 23307 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a9b1b5ca75d43308184b1105a3fee03: Bootstrap starting.
I20260924 10:01:20.288395 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 10:01:20.289525 23307 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7a9b1b5ca75d43308184b1105a3fee03: Neither blocks nor log segments found. Creating new log.
I20260924 10:01:20.290568 23307 log.cc:824] T 00000000000000000000000000000000 P 7a9b1b5ca75d43308184b1105a3fee03: Log is configured to *not* fsync() on all Append() calls
I20260924 10:01:20.293215 23307 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a9b1b5ca75d43308184b1105a3fee03: No bootstrap required, opened a new log
I20260924 10:01:20.290617 23134 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "1bdfe206f9fe4614ac8319ee70514d24"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56972
I20260924 10:01:20.300132 23307 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7a9b1b5ca75d43308184b1105a3fee03 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bdfe206f9fe4614ac8319ee70514d24" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38567 } } peers { permanent_uuid: "31c2308dcf0147c8b752cb0f466f95f5" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46191 } } peers { permanent_uuid: "7a9b1b5ca75d43308184b1105a3fee03" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44999 } }
I20260924 10:01:20.300479 23307 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7a9b1b5ca75d43308184b1105a3fee03 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a9b1b5ca75d43308184b1105a3fee03, State: Initialized, Role: FOLLOWER
I20260924 10:01:20.300931 23307 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7a9b1b5ca75d43308184b1105a3fee03 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bdfe206f9fe4614ac8319ee70514d24" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38567 } } peers { permanent_uuid: "31c2308dcf0147c8b752cb0f466f95f5" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46191 } } peers { permanent_uuid: "7a9b1b5ca75d43308184b1105a3fee03" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44999 } }
I20260924 10:01:20.301760 23313 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a9b1b5ca75d43308184b1105a3fee03 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bdfe206f9fe4614ac8319ee70514d24" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38567 } } peers { permanent_uuid: "31c2308dcf0147c8b752cb0f466f95f5" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46191 } } peers { permanent_uuid: "7a9b1b5ca75d43308184b1105a3fee03" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44999 } } }
I20260924 10:01:20.302065 23313 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a9b1b5ca75d43308184b1105a3fee03 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 10:01:20.303013 23307 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7a9b1b5ca75d43308184b1105a3fee03 [sys.catalog]: configured and running, proceeding with master startup.
W20260924 10:01:20.312106 23324 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7a9b1b5ca75d43308184b1105a3fee03: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 10:01:20.312240 23324 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 10:01:20.309654 23240 master_runner.cc:186] Error getting master registration for 127.0.97.62:38567: OK, instance_id { permanent_uuid: "1bdfe206f9fe4614ac8319ee70514d24" instance_seqno: 1790244079252828 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 10:01:20.320710 23240 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 10:01:20.387773 23159 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44999 } attempt: 4
Sep 24 10:01:20 dist-test-slave-w8d2 krb5kdc[23079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244079, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260924 10:01:20.404289 23310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:20.388593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41236 (local address 127.0.97.60:44999)
0924 10:01:20.388811 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0924 10:01:20.388817 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:01:20.388886 (+ 69us) server_negotiation.cc:407] Connection header received
0924 10:01:20.389065 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:01:20.389071 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:01:20.389157 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:01:20.389260 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:01:20.390007 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:20.391177 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:20.392913 (+ 1736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:20.393601 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:20.393718 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:01:20.397695 (+ 3977us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:01:20.397702 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:01:20.397708 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:01:20.397750 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:01:20.401329 (+ 3579us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:01:20.402126 (+ 797us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:01:20.402131 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:01:20.402150 (+ 19us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:01:20.402286 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:01:20.402697 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:01:20.402701 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:01:20.402705 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:01:20.403073 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:01:20.403214 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0924 10:01:20.403703 (+ 489us) server_negotiation.cc:299] Negotiation successful
0924 10:01:20.403935 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 10:01:20.404314 23162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:20.388545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44999 (local address 127.0.0.1:41236)
0924 10:01:20.388759 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0924 10:01:20.388795 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 10:01:20.388941 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:01:20.389331 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:01:20.389335 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:01:20.389712 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:01:20.389874 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:01:20.389883 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:01:20.391338 (+ 1455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:01:20.391342 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:01:20.392753 (+ 1411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:01:20.392768 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:01:20.393747 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:01:20.393752 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:01:20.393985 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:01:20.394634 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:01:20.394655 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:01:20.397471 (+ 2816us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:01:20.401474 (+ 4003us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:01:20.401479 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:01:20.401484 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:01:20.402014 (+ 530us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:01:20.402394 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:01:20.402399 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:01:20.402403 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:01:20.402577 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:01:20.403179 (+ 602us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:01:20.403184 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:01:20.403401 (+ 217us) client_negotiation.cc:769] Sending connection context
0924 10:01:20.403567 (+ 166us) client_negotiation.cc:240] Negotiation successful
0924 10:01:20.403932 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260924 10:01:20.410179 23159 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1bdfe206f9fe4614ac8319ee70514d24: Bootstrap starting.
I20260924 10:01:20.414222 23159 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1bdfe206f9fe4614ac8319ee70514d24: Neither blocks nor log segments found. Creating new log.
I20260924 10:01:20.415582 23159 log.cc:824] T 00000000000000000000000000000000 P 1bdfe206f9fe4614ac8319ee70514d24: Log is configured to *not* fsync() on all Append() calls
I20260924 10:01:20.418849 23159 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1bdfe206f9fe4614ac8319ee70514d24: No bootstrap required, opened a new log
I20260924 10:01:20.426424 23159 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1bdfe206f9fe4614ac8319ee70514d24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bdfe206f9fe4614ac8319ee70514d24" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38567 } } peers { permanent_uuid: "31c2308dcf0147c8b752cb0f466f95f5" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46191 } } peers { permanent_uuid: "7a9b1b5ca75d43308184b1105a3fee03" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44999 } }
I20260924 10:01:20.426828 23159 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1bdfe206f9fe4614ac8319ee70514d24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1bdfe206f9fe4614ac8319ee70514d24, State: Initialized, Role: FOLLOWER
I20260924 10:01:20.427381 23159 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1bdfe206f9fe4614ac8319ee70514d24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bdfe206f9fe4614ac8319ee70514d24" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38567 } } peers { permanent_uuid: "31c2308dcf0147c8b752cb0f466f95f5" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46191 } } peers { permanent_uuid: "7a9b1b5ca75d43308184b1105a3fee03" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44999 } }
I20260924 10:01:20.428376 23327 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1bdfe206f9fe4614ac8319ee70514d24 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bdfe206f9fe4614ac8319ee70514d24" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38567 } } peers { permanent_uuid: "31c2308dcf0147c8b752cb0f466f95f5" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46191 } } peers { permanent_uuid: "7a9b1b5ca75d43308184b1105a3fee03" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44999 } } }
I20260924 10:01:20.428817 23327 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1bdfe206f9fe4614ac8319ee70514d24 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 10:01:20.429724 23159 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1bdfe206f9fe4614ac8319ee70514d24 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:01:20.429795 23134 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1bdfe206f9fe4614ac8319ee70514d24 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 10:01:20.429988 23134 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1bdfe206f9fe4614ac8319ee70514d24 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:01:20.432619 23134 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1bdfe206f9fe4614ac8319ee70514d24 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bdfe206f9fe4614ac8319ee70514d24" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38567 } } peers { permanent_uuid: "31c2308dcf0147c8b752cb0f466f95f5" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46191 } } peers { permanent_uuid: "7a9b1b5ca75d43308184b1105a3fee03" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44999 } }
I20260924 10:01:20.434297 23134 leader_election.cc:288] T 00000000000000000000000000000000 P 1bdfe206f9fe4614ac8319ee70514d24 [CANDIDATE]: Term 1 election: Requested vote from peers 31c2308dcf0147c8b752cb0f466f95f5 (127.0.97.61:46191), 7a9b1b5ca75d43308184b1105a3fee03 (127.0.97.60:44999)
I20260924 10:01:20.434713 23282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1bdfe206f9fe4614ac8319ee70514d24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "7a9b1b5ca75d43308184b1105a3fee03"
I20260924 10:01:20.435213 23282 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7a9b1b5ca75d43308184b1105a3fee03 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:01:20.438654 23282 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7a9b1b5ca75d43308184b1105a3fee03 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1bdfe206f9fe4614ac8319ee70514d24 in term 1.
I20260924 10:01:20.439429 23110 leader_election.cc:302] T 00000000000000000000000000000000 P 1bdfe206f9fe4614ac8319ee70514d24 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1bdfe206f9fe4614ac8319ee70514d24, 7a9b1b5ca75d43308184b1105a3fee03; no voters:
I20260924 10:01:20.439950 23327 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1bdfe206f9fe4614ac8319ee70514d24 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:01:20.440570 23233 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44999 } attempt: 5
Sep 24 10:01:20 dist-test-slave-w8d2 krb5kdc[23079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244079, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260924 10:01:20.452680 23327 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1bdfe206f9fe4614ac8319ee70514d24 [term 1 LEADER]: Becoming Leader. State: Replica: 1bdfe206f9fe4614ac8319ee70514d24, State: Running, Role: LEADER
W20260924 10:01:20.453379 23261 tablet.cc:2369] T 00000000000000000000000000000000 P 7a9b1b5ca75d43308184b1105a3fee03: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 10:01:20.453433 23327 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1bdfe206f9fe4614ac8319ee70514d24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bdfe206f9fe4614ac8319ee70514d24" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38567 } } peers { permanent_uuid: "31c2308dcf0147c8b752cb0f466f95f5" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46191 } } peers { permanent_uuid: "7a9b1b5ca75d43308184b1105a3fee03" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44999 } }
I20260924 10:01:20.457717 23335 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1bdfe206f9fe4614ac8319ee70514d24 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1bdfe206f9fe4614ac8319ee70514d24. Latest consensus state: current_term: 1 leader_uuid: "1bdfe206f9fe4614ac8319ee70514d24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bdfe206f9fe4614ac8319ee70514d24" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38567 } } peers { permanent_uuid: "31c2308dcf0147c8b752cb0f466f95f5" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46191 } } peers { permanent_uuid: "7a9b1b5ca75d43308184b1105a3fee03" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44999 } } }
I20260924 10:01:20.458180 23335 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1bdfe206f9fe4614ac8319ee70514d24 [sys.catalog]: This master's current role is: LEADER
I20260924 10:01:20.458369 23235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:20.441180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44999 (local address 127.0.0.1:41248)
0924 10:01:20.441544 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0924 10:01:20.441562 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 10:01:20.441685 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:01:20.442565 (+ 880us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:01:20.442570 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:01:20.443128 (+ 558us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:01:20.443363 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:01:20.443376 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:01:20.444839 (+ 1463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:01:20.444845 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:01:20.446311 (+ 1466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:01:20.446328 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:01:20.447529 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:01:20.447535 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:01:20.447730 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:01:20.448382 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:01:20.448408 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:01:20.451354 (+ 2946us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:01:20.455383 (+ 4029us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:01:20.455389 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:01:20.455393 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:01:20.455950 (+ 557us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:01:20.456392 (+ 442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:01:20.456397 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:01:20.456401 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:01:20.456579 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:01:20.457301 (+ 722us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:01:20.457307 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:01:20.457478 (+ 171us) client_negotiation.cc:769] Sending connection context
0924 10:01:20.457636 (+ 158us) client_negotiation.cc:240] Negotiation successful
0924 10:01:20.457971 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227}
I20260924 10:01:20.459384 23310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:20.441910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41248 (local address 127.0.97.60:44999)
0924 10:01:20.442086 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0924 10:01:20.442092 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:01:20.442111 (+ 19us) server_negotiation.cc:407] Connection header received
0924 10:01:20.442170 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:01:20.442175 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:01:20.442254 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:01:20.442348 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:01:20.443560 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:20.444651 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:20.446495 (+ 1844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:20.447187 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:20.447322 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:01:20.451595 (+ 4273us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:01:20.451603 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:01:20.451608 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:01:20.451654 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:01:20.455181 (+ 3527us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:01:20.456091 (+ 910us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:01:20.456097 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:01:20.456100 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:01:20.456250 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:01:20.456703 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:01:20.456708 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:01:20.456711 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:01:20.457088 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:01:20.457234 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0924 10:01:20.458763 (+ 1529us) server_negotiation.cc:299] Negotiation successful
0924 10:01:20.459021 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 10:01:20.465426 23233 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31c2308dcf0147c8b752cb0f466f95f5: Bootstrap starting.
I20260924 10:01:20.469858 23233 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 31c2308dcf0147c8b752cb0f466f95f5: Neither blocks nor log segments found. Creating new log.
I20260924 10:01:20.483635 23233 log.cc:824] T 00000000000000000000000000000000 P 31c2308dcf0147c8b752cb0f466f95f5: Log is configured to *not* fsync() on all Append() calls
I20260924 10:01:20.490763 23233 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31c2308dcf0147c8b752cb0f466f95f5: No bootstrap required, opened a new log
W20260924 10:01:20.493487 23113 tablet.cc:2369] T 00000000000000000000000000000000 P 1bdfe206f9fe4614ac8319ee70514d24: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 10:01:20.498032 23341 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:01:20.515205 23233 raft_consensus.cc:348] T 00000000000000000000000000000000 P 31c2308dcf0147c8b752cb0f466f95f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bdfe206f9fe4614ac8319ee70514d24" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38567 } } peers { permanent_uuid: "31c2308dcf0147c8b752cb0f466f95f5" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46191 } } peers { permanent_uuid: "7a9b1b5ca75d43308184b1105a3fee03" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44999 } }
I20260924 10:01:20.515825 23233 raft_consensus.cc:729] T 00000000000000000000000000000000 P 31c2308dcf0147c8b752cb0f466f95f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31c2308dcf0147c8b752cb0f466f95f5, State: Initialized, Role: FOLLOWER
I20260924 10:01:20.516419 23233 consensus_queue.cc:260] T 00000000000000000000000000000000 P 31c2308dcf0147c8b752cb0f466f95f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bdfe206f9fe4614ac8319ee70514d24" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38567 } } peers { permanent_uuid: "31c2308dcf0147c8b752cb0f466f95f5" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46191 } } peers { permanent_uuid: "7a9b1b5ca75d43308184b1105a3fee03" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44999 } }
I20260924 10:01:20.524675 23350 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31c2308dcf0147c8b752cb0f466f95f5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bdfe206f9fe4614ac8319ee70514d24" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38567 } } peers { permanent_uuid: "31c2308dcf0147c8b752cb0f466f95f5" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46191 } } peers { permanent_uuid: "7a9b1b5ca75d43308184b1105a3fee03" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44999 } } }
I20260924 10:01:20.525125 23350 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31c2308dcf0147c8b752cb0f466f95f5 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 10:01:20.525792 23237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:20.467564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35300 (local address 127.0.97.61:46191)
0924 10:01:20.467738 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0924 10:01:20.467744 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:01:20.502148 (+ 34404us) server_negotiation.cc:407] Connection header received
0924 10:01:20.502488 (+ 340us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:01:20.502493 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:01:20.502569 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:01:20.502634 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:01:20.507328 (+ 4694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:20.508444 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:20.511401 (+ 2957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:20.512072 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:20.512188 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:01:20.518123 (+ 5935us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:01:20.518130 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:01:20.518135 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:01:20.518183 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:01:20.521658 (+ 3475us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:01:20.522911 (+ 1253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:01:20.522917 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:01:20.522921 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:01:20.523081 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:01:20.523717 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:01:20.523722 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:01:20.523726 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:01:20.524084 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:01:20.524331 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0924 10:01:20.525170 (+ 839us) server_negotiation.cc:299] Negotiation successful
0924 10:01:20.525425 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 10:01:20.527076 23233 sys_catalog.cc:564] T 00000000000000000000000000000000 P 31c2308dcf0147c8b752cb0f466f95f5 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:01:20.527130 23208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1bdfe206f9fe4614ac8319ee70514d24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "31c2308dcf0147c8b752cb0f466f95f5"
I20260924 10:01:20.527639 23208 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 31c2308dcf0147c8b752cb0f466f95f5 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:01:20.530989 23208 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 31c2308dcf0147c8b752cb0f466f95f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1bdfe206f9fe4614ac8319ee70514d24 in term 1.
I20260924 10:01:20.537320 23341 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:01:20.540107 23163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:20.462122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56988 (local address 127.0.97.62:38567)
0924 10:01:20.462422 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0924 10:01:20.462428 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:01:20.501854 (+ 39426us) server_negotiation.cc:407] Connection header received
0924 10:01:20.502288 (+ 434us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:01:20.502295 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:01:20.502397 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:01:20.502472 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:01:20.506650 (+ 4178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:20.509116 (+ 2466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:20.514329 (+ 5213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:20.515085 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:20.515210 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:01:20.524688 (+ 9478us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:01:20.524695 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:01:20.524701 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:01:20.524756 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:01:20.530047 (+ 5291us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:01:20.537548 (+ 7501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:01:20.537554 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:01:20.537557 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:01:20.537765 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:01:20.538218 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:01:20.538224 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:01:20.538228 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:01:20.538625 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:01:20.538864 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0924 10:01:20.539484 (+ 620us) server_negotiation.cc:299] Negotiation successful
0924 10:01:20.539747 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260924 10:01:20.544821 23310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:20.465200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41260 (local address 127.0.97.60:44999)
0924 10:01:20.471227 (+ 6027us) server_negotiation.cc:206] Beginning negotiation
0924 10:01:20.471261 (+ 34us) server_negotiation.cc:399] Waiting for connection header
0924 10:01:20.501853 (+ 30592us) server_negotiation.cc:407] Connection header received
0924 10:01:20.502978 (+ 1125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:01:20.502983 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:01:20.503094 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:01:20.503171 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:01:20.516196 (+ 13025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:20.517223 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:20.523342 (+ 6119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:20.523989 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:20.524088 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:01:20.533414 (+ 9326us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:01:20.533421 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:01:20.533426 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:01:20.533469 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:01:20.536879 (+ 3410us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:01:20.542287 (+ 5408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:01:20.542294 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:01:20.542298 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:01:20.542470 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:01:20.542973 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:01:20.542979 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:01:20.542983 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:01:20.543387 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:01:20.543631 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0924 10:01:20.544136 (+ 505us) server_negotiation.cc:299] Negotiation successful
0924 10:01:20.544415 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5899,"thread_start_us":19390,"threads_started":1}
I20260924 10:01:20.552353 23282 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7a9b1b5ca75d43308184b1105a3fee03 [term 1 FOLLOWER]: Refusing update from remote peer 1bdfe206f9fe4614ac8319ee70514d24: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:01:20.552428 23208 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 31c2308dcf0147c8b752cb0f466f95f5 [term 1 FOLLOWER]: Refusing update from remote peer 1bdfe206f9fe4614ac8319ee70514d24: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:01:20.553243 23335 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1bdfe206f9fe4614ac8319ee70514d24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a9b1b5ca75d43308184b1105a3fee03" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:01:20.553651 23327 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1bdfe206f9fe4614ac8319ee70514d24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31c2308dcf0147c8b752cb0f466f95f5" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:01:20.554951 23092 master_runner.cc:186] Error getting master registration for 127.0.97.61:46191: OK, instance_id { permanent_uuid: "31c2308dcf0147c8b752cb0f466f95f5" instance_seqno: 1790244079722719 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 10:01:20.571420 23092 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 10:01:20.574414 23350 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31c2308dcf0147c8b752cb0f466f95f5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1bdfe206f9fe4614ac8319ee70514d24. Latest consensus state: current_term: 1 leader_uuid: "1bdfe206f9fe4614ac8319ee70514d24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bdfe206f9fe4614ac8319ee70514d24" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38567 } } peers { permanent_uuid: "31c2308dcf0147c8b752cb0f466f95f5" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46191 } } peers { permanent_uuid: "7a9b1b5ca75d43308184b1105a3fee03" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44999 } } }
I20260924 10:01:20.574664 23350 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31c2308dcf0147c8b752cb0f466f95f5 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 10:01:20.576130 23368 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260924 10:01:20.578498 23359 mvcc.cc:205] Tried to move back new op lower bound from 7332839753933340672 to 7332839753544814592. Current Snapshot: MvccSnapshot[applied={T|T < 7332839753933340672}]
I20260924 10:01:20.580312 23335 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1bdfe206f9fe4614ac8319ee70514d24 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1bdfe206f9fe4614ac8319ee70514d24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bdfe206f9fe4614ac8319ee70514d24" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38567 } } peers { permanent_uuid: "31c2308dcf0147c8b752cb0f466f95f5" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46191 } } peers { permanent_uuid: "7a9b1b5ca75d43308184b1105a3fee03" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44999 } } }
I20260924 10:01:20.580580 23335 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1bdfe206f9fe4614ac8319ee70514d24 [sys.catalog]: This master's current role is: LEADER
I20260924 10:01:20.580519 23313 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a9b1b5ca75d43308184b1105a3fee03 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1bdfe206f9fe4614ac8319ee70514d24. Latest consensus state: current_term: 1 leader_uuid: "1bdfe206f9fe4614ac8319ee70514d24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bdfe206f9fe4614ac8319ee70514d24" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38567 } } peers { permanent_uuid: "31c2308dcf0147c8b752cb0f466f95f5" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46191 } } peers { permanent_uuid: "7a9b1b5ca75d43308184b1105a3fee03" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44999 } } }
I20260924 10:01:20.580824 23313 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a9b1b5ca75d43308184b1105a3fee03 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 10:01:20.582000 23327 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1bdfe206f9fe4614ac8319ee70514d24 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1bdfe206f9fe4614ac8319ee70514d24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bdfe206f9fe4614ac8319ee70514d24" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38567 } } peers { permanent_uuid: "31c2308dcf0147c8b752cb0f466f95f5" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46191 } } peers { permanent_uuid: "7a9b1b5ca75d43308184b1105a3fee03" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44999 } } }
I20260924 10:01:20.582273 23327 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1bdfe206f9fe4614ac8319ee70514d24 [sys.catalog]: This master's current role is: LEADER
I20260924 10:01:20.582267 23362 mvcc.cc:205] Tried to move back new op lower bound from 7332839753933340672 to 7332839753544814592. Current Snapshot: MvccSnapshot[applied={T|T < 7332839753933340672}]
I20260924 10:01:20.584215 23341 catalog_manager.cc:1347] Generated new cluster ID: 75698c2318fb4a76803967734345e9aa
I20260924 10:01:20.584303 23341 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:01:20.586958 23361 mvcc.cc:205] Tried to move back new op lower bound from 7332839753933340672 to 7332839753544814592. Current Snapshot: MvccSnapshot[applied={T|T < 7332839753933340672}]
I20260924 10:01:20.604592 23313 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a9b1b5ca75d43308184b1105a3fee03 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1bdfe206f9fe4614ac8319ee70514d24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bdfe206f9fe4614ac8319ee70514d24" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38567 } } peers { permanent_uuid: "31c2308dcf0147c8b752cb0f466f95f5" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46191 } } peers { permanent_uuid: "7a9b1b5ca75d43308184b1105a3fee03" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44999 } } }
I20260924 10:01:20.604869 23313 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a9b1b5ca75d43308184b1105a3fee03 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 10:01:20.608143 23350 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31c2308dcf0147c8b752cb0f466f95f5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1bdfe206f9fe4614ac8319ee70514d24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bdfe206f9fe4614ac8319ee70514d24" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38567 } } peers { permanent_uuid: "31c2308dcf0147c8b752cb0f466f95f5" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46191 } } peers { permanent_uuid: "7a9b1b5ca75d43308184b1105a3fee03" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44999 } } }
I20260924 10:01:20.608371 23350 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31c2308dcf0147c8b752cb0f466f95f5 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 10:01:20.628971 23341 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:01:20.632012 23341 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:01:20.648451 23341 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1bdfe206f9fe4614ac8319ee70514d24: Generated new TSK 0
I20260924 10:01:20.649590 23341 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 10:01:20.725827 23163 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:01:20.717266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56992 (local address 127.0.97.62:38567)
0924 10:01:20.717469 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0924 10:01:20.717476 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:01:20.723768 (+ 6292us) server_negotiation.cc:407] Connection header received
0924 10:01:20.724076 (+ 308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:01:20.724082 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:01:20.724196 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:01:20.724274 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:01:20.725558 (+ 1284us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56992: BlockingRecv error: recv got EOF from 127.0.0.1:56992 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
W20260924 10:01:20.729637 23310 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:01:20.717391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41262 (local address 127.0.97.60:44999)
0924 10:01:20.717571 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0924 10:01:20.717577 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:01:20.726469 (+ 8892us) server_negotiation.cc:407] Connection header received
0924 10:01:20.727385 (+ 916us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:01:20.727392 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:01:20.727522 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:01:20.727598 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:01:20.729336 (+ 1738us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41262: BlockingRecv error: recv got EOF from 127.0.0.1:41262 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
W20260924 10:01:20.735333 23237 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:01:20.723479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35304 (local address 127.0.97.61:46191)
0924 10:01:20.723656 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0924 10:01:20.723662 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:01:20.730414 (+ 6752us) server_negotiation.cc:407] Connection header received
0924 10:01:20.731288 (+ 874us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:01:20.731294 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:01:20.731376 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:01:20.731448 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:01:20.735066 (+ 3618us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35304: BlockingRecv error: recv got EOF from 127.0.0.1:35304 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
W20260924 10:01:20.776880 23237 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:01:20.771008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35310 (local address 127.0.97.61:46191)
0924 10:01:20.771227 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0924 10:01:20.771258 (+ 31us) server_negotiation.cc:399] Waiting for connection header
0924 10:01:20.774913 (+ 3655us) server_negotiation.cc:407] Connection header received
0924 10:01:20.775323 (+ 410us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:01:20.775330 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:01:20.775477 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:01:20.775563 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:01:20.776623 (+ 1060us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35310: BlockingRecv error: recv got EOF from 127.0.0.1:35310 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260924 10:01:20.789390 23237 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:01:20.785025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35314 (local address 127.0.97.61:46191)
0924 10:01:20.785214 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0924 10:01:20.785219 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:01:20.787538 (+ 2319us) server_negotiation.cc:407] Connection header received
0924 10:01:20.787878 (+ 340us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:01:20.787884 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:01:20.787978 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:01:20.788048 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:01:20.789166 (+ 1118us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35314: BlockingRecv error: recv got EOF from 127.0.0.1:35314 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260924 10:01:20.798592 23237 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:01:20.796500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35330 (local address 127.0.97.61:46191)
0924 10:01:20.796672 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0924 10:01:20.796678 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:01:20.797144 (+ 466us) server_negotiation.cc:407] Connection header received
0924 10:01:20.797343 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:01:20.797348 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:01:20.797451 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:01:20.797559 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:01:20.798354 (+ 795us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35330: BlockingRecv error: recv got EOF from 127.0.0.1:35330 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
W20260924 10:01:20.808145 23237 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:01:20.806215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35342 (local address 127.0.97.61:46191)
0924 10:01:20.806434 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0924 10:01:20.806440 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:01:20.806761 (+ 321us) server_negotiation.cc:407] Connection header received
0924 10:01:20.806935 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:01:20.806941 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:01:20.807030 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:01:20.807132 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:01:20.807921 (+ 789us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35342: BlockingRecv error: recv got EOF from 127.0.0.1:35342 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
W20260924 10:01:20.822551 23237 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:01:20.820313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35350 (local address 127.0.97.61:46191)
0924 10:01:20.820513 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0924 10:01:20.820519 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:01:20.821049 (+ 530us) server_negotiation.cc:407] Connection header received
0924 10:01:20.821248 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:01:20.821255 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:01:20.821347 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:01:20.821460 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:01:20.822317 (+ 857us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35350: BlockingRecv error: recv got EOF from 127.0.0.1:35350 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
W20260924 10:01:20.846196 23237 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:01:20.843941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35366 (local address 127.0.97.61:46191)
0924 10:01:20.844137 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0924 10:01:20.844143 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:01:20.844703 (+ 560us) server_negotiation.cc:407] Connection header received
0924 10:01:20.844918 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:01:20.844925 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:01:20.845023 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:01:20.845129 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:01:20.845955 (+ 826us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35366: BlockingRecv error: recv got EOF from 127.0.0.1:35366 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
W20260924 10:01:20.883765 23237 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:01:20.881523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35368 (local address 127.0.97.61:46191)
0924 10:01:20.881707 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0924 10:01:20.881713 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:01:20.882248 (+ 535us) server_negotiation.cc:407] Connection header received
0924 10:01:20.882436 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:01:20.882442 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:01:20.882545 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:01:20.882667 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:01:20.883479 (+ 812us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35368: BlockingRecv error: recv got EOF from 127.0.0.1:35368 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
W20260924 10:01:20.952484 23237 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:01:20.950732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35378 (local address 127.0.97.61:46191)
0924 10:01:20.950944 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0924 10:01:20.950950 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:01:20.951204 (+ 254us) server_negotiation.cc:407] Connection header received
0924 10:01:20.951397 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:01:20.951403 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:01:20.951500 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:01:20.951620 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:01:20.952274 (+ 654us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35378: BlockingRecv error: recv got EOF from 127.0.0.1:35378 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
W20260924 10:01:21.086416 23237 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:01:21.084755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35392 (local address 127.0.97.61:46191)
0924 10:01:21.084981 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0924 10:01:21.084987 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:01:21.085245 (+ 258us) server_negotiation.cc:407] Connection header received
0924 10:01:21.085404 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:01:21.085408 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:01:21.085484 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:01:21.085570 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:01:21.086199 (+ 629us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35392: BlockingRecv error: recv got EOF from 127.0.0.1:35392 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
I20260924 10:01:21.314263 23324 catalog_manager.cc:1259] Loaded cluster ID: 75698c2318fb4a76803967734345e9aa
I20260924 10:01:21.314370 23324 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 7a9b1b5ca75d43308184b1105a3fee03: loading cluster ID for follower catalog manager: success
I20260924 10:01:21.316841 23324 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 7a9b1b5ca75d43308184b1105a3fee03: acquiring CA information for follower catalog manager: success
I20260924 10:01:21.318590 23324 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 7a9b1b5ca75d43308184b1105a3fee03: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260924 10:01:21.350046 23237 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:01:21.348429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35396 (local address 127.0.97.61:46191)
0924 10:01:21.348639 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0924 10:01:21.348653 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 10:01:21.348851 (+ 198us) server_negotiation.cc:407] Connection header received
0924 10:01:21.348998 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:01:21.349002 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:01:21.349075 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:01:21.349158 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:01:21.349840 (+ 682us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35396: BlockingRecv error: recv got EOF from 127.0.0.1:35396 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
I20260924 10:01:21.580250 23368 catalog_manager.cc:1259] Loaded cluster ID: 75698c2318fb4a76803967734345e9aa
I20260924 10:01:21.580359 23368 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 31c2308dcf0147c8b752cb0f466f95f5: loading cluster ID for follower catalog manager: success
I20260924 10:01:21.583148 23368 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 31c2308dcf0147c8b752cb0f466f95f5: acquiring CA information for follower catalog manager: success
I20260924 10:01:21.584987 23368 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 31c2308dcf0147c8b752cb0f466f95f5: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 10:01:21.881556 23437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:21.874602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35398 (local address 127.0.97.61:46191)
0924 10:01:21.875212 (+ 610us) server_negotiation.cc:206] Beginning negotiation
0924 10:01:21.875218 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:01:21.875531 (+ 313us) server_negotiation.cc:407] Connection header received
0924 10:01:21.875727 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:01:21.875730 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:01:21.875877 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:01:21.875983 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:01:21.876447 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:21.877298 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:21.879186 (+ 1888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:21.879671 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:21.879764 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:01:21.880360 (+ 596us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:01:21.880921 (+ 561us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:01:21.881028 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0924 10:01:21.881153 (+ 125us) server_negotiation.cc:299] Negotiation successful
0924 10:01:21.881228 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":450,"thread_start_us":276,"threads_started":1}
I20260924 10:01:21.893888 23443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:21.888238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41276 (local address 127.0.97.60:44999)
0924 10:01:21.888974 (+ 736us) server_negotiation.cc:206] Beginning negotiation
0924 10:01:21.888980 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:01:21.888997 (+ 17us) server_negotiation.cc:407] Connection header received
0924 10:01:21.889088 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:01:21.889091 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:01:21.889232 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:01:21.889375 (+ 143us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:01:21.889737 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:21.890445 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:21.891772 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:21.892167 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:21.892272 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:01:21.892735 (+ 463us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:01:21.893254 (+ 519us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:01:21.893389 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0924 10:01:21.893477 (+ 88us) server_negotiation.cc:299] Negotiation successful
0924 10:01:21.893566 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":619,"thread_start_us":514,"threads_started":1}
I20260924 10:01:21.914839 23443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:21.900483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41278 (local address 127.0.97.60:44999)
0924 10:01:21.905919 (+ 5436us) server_negotiation.cc:206] Beginning negotiation
0924 10:01:21.905927 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 10:01:21.908517 (+ 2590us) server_negotiation.cc:407] Connection header received
0924 10:01:21.908665 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:01:21.908671 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:01:21.908772 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:01:21.908847 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:01:21.909540 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:21.910565 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:21.912538 (+ 1973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:21.913154 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:21.913262 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:01:21.913816 (+ 554us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:01:21.913938 (+ 122us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:01:21.914008 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 10:01:21.914461 (+ 453us) server_negotiation.cc:299] Negotiation successful
0924 10:01:21.914558 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5263}
I20260924 10:01:21.920418 23437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:21.899928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35408 (local address 127.0.97.61:46191)
0924 10:01:21.900096 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0924 10:01:21.900101 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:01:21.915765 (+ 15664us) server_negotiation.cc:407] Connection header received
0924 10:01:21.915901 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:01:21.915906 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:01:21.915985 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:01:21.916066 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:01:21.916727 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:21.917417 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:21.918864 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:21.919296 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:21.919407 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:01:21.919775 (+ 368us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:01:21.919875 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:01:21.919946 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 10:01:21.920076 (+ 130us) server_negotiation.cc:299] Negotiation successful
0924 10:01:21.920154 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 10:01:21.921120 23449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:01:21.899869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57008 (local address 127.0.97.62:38567)
0924 10:01:21.915556 (+ 15687us) server_negotiation.cc:206] Beginning negotiation
0924 10:01:21.915563 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:01:21.915582 (+ 19us) server_negotiation.cc:407] Connection header received
0924 10:01:21.915683 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:01:21.915686 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:01:21.915897 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:01:21.916226 (+ 329us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:01:21.916817 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:21.917563 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:21.919023 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:01:21.919459 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:01:21.919566 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:01:21.920003 (+ 437us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:01:21.920380 (+ 377us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:01:21.920469 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0924 10:01:21.920829 (+ 360us) server_negotiation.cc:299] Negotiation successful
0924 10:01:21.920910 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15560,"thread_start_us":16172,"threads_started":1}
I20260924 10:01:21.925352 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu pid 23092
I20260924 10:01:21.926306 23154 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:01:22.033727 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu with pid 23092
W20260924 10:01:22.050127 23185 connection.cc:535] server connection from 127.0.0.1:35268 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260924 10:01:22.050451 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu pid 23166
I20260924 10:01:22.051486 23228 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:01:22.155081 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu with pid 23166
I20260924 10:01:22.171074 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu pid 23240
I20260924 10:01:22.172151 23302 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:01:22.277405 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_ZmK/build/asan/bin/kudu with pid 23240
2026-09-24T10:01:22Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (4684 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (8385 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (346604 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST